research article a motion detection algorithm using local...
TRANSCRIPT
Research ArticleA Motion Detection Algorithm Using Local Phase Information
Aurel A Lazar Nikul H Ukani and Yiyin Zhou
Department of Electrical Engineering Columbia University New York NY 10027 USA
Correspondence should be addressed to Aurel A Lazar aureleecolumbiaedu
Received 15 June 2015 Revised 14 October 2015 Accepted 15 October 2015
Academic Editor Ezequiel Lopez-Rubio
Copyright copy 2016 Aurel A Lazar et al This is an open access article distributed under the Creative Commons Attribution Licensewhich permits unrestricted use distribution and reproduction in any medium provided the original work is properly cited
Previous research demonstrated that global phase alone can be used to faithfully represent visual scenes Here we provide areconstruction algorithm by using only local phase information We also demonstrate that local phase alone can be effectively usedto detect local motion The local phase-based motion detector is akin to models employed to detect motion in biological visionfor example the Reichardt detector The local phase-based motion detection algorithm introduced here consists of two buildingblocks The first building block measuresevaluates the temporal change of the local phase The temporal derivative of the localphase is shown to exhibit the structure of a second order Volterra kernel with two normalized inputs We provide an efficient FFT-based algorithm for implementing the change of the local phase The second processing building block implements the detector itcompares the maximum of the Radon transform of the local phase derivative with a chosen threshold We demonstrate examplesof applying the local phase-based motion detection algorithm on several video sequences We also show how the locally detectedmotion can be used for segmenting moving objects in video scenes and compare our local phase-based algorithm to segmentationachieved with a widely used optic flow algorithm
1 Introduction
Following Marr the design of an information processingsystem can be approached on multiple levels [1] Figure 1illustrates two levels of abstraction namely the algorithmiclevel and the physical circuit level On the algorithmic levelone studies procedurally how the information is processedindependently of the physical realization The circuit levelconcerns the actual realization of the algorithm in physicalhardware for example a biological neural circuit or siliconcircuits in a digital signal processor In this paper we putforth a simple motion detection algorithm that is inspired bymotion detection models of biological visual systems (in vivoneural circuit) and provide an efficient realization that caneasily be implemented on commodity (in silico) DSP chips
Visual motion detection is critical to the survival ofanimals Many biological visual systems have evolved highlyefficienteffective neural circuits to detect visual motionMotion detection is performed in parallel with other visualcoding circuits and starts already in the early stages of visualprocessing In the retina of vertebrates it is known that at least
three types of Direction-Selective Ganglion Cells (DSGC) areresponsible for signaling visual motion at this early stage [2]In flies direction-selective neurons are found in the opticlobe 3 synapses away from the photoreceptors [3]
The small number of synapses between photoreceptorsand direction-selective neurons suggests that the processinginvolved in motion detection is not highly complex but stillvery effective In addition the biological motion detectioncircuits are organized in a highly parallel way to enable fastconcurrent computation of motion It is also interesting tonote that the early stages of motion detection are carriedout largely in the absence of spiking neurons indicating thatinitial stages of motion detection are preferably performed inthe ldquoanalogrdquo domain Taking advantage of continuous timeprocessing may be critical for quickly processing motionsince motion intrinsically elicits fast and large changes in theintensity levels that is large amounts of data under stringenttime constraints
Modern computer-based motion detection algorithmsoften employ optic flow techniques to estimate spatialchanges in consecutive image frames [4 5] Although often
Hindawi Publishing CorporationComputational Intelligence and NeuroscienceVolume 2016 Article ID 7915245 20 pageshttpdxdoiorg10115520167915245
2 Computational Intelligence and Neuroscience
Algorithm
In vivo neural circuit
In silicoDSP chip
Figure 1 Algorithms can have different physical realizations Forexample an algorithm can be implemented by neural circuits in abiological system Alternately it can be implemented on a digitalsignal processor Algorithms direct the implementation on thephysical layer Conversely biological neural circuits inspire newalgorithmic designs which can in turn be expressed and improvedupon by a realization in silico
time optic flow estimation algorithms produce accurateresults the computational demand to perform many of thesealgorithms is too high for real-time implementation
Several models for biological motion detection are avail-able and their architecture is quite simple [6] The Reichardtmotion detector [7] was thought to be the underlyingmodel for motion detection in insects [8] The model isbased on a correlation method to extract motion inducedby spatiotemporal information patterns of light intensityTherefore it relies on a correlationmultiplication operationA second model is the motion energy detector [9] It usesspatiotemporal separable filters and a squaring nonlinearityto compute motion energy and it was shown to be equivalentto the Reichardt motion detector Earlier work in the rabbitretina was the foundation to the Barlow-Levick model [10]of motion detection The model relies on inhibition tocompensate motion in the null direction
In this paper we provide an alternative motion detectionalgorithm based on local phase information of the visualscene Similar to mechanisms in other biological models itoperates in continuous time and in parallel Moreover themotion detection algorithm we propose can be efficientlyimplemented on parallel hardware This is again similar tothe properties of biological motion detection systems Ratherthan focusing on velocity of motion we focus on localizationthat is where the motion occurs in the visual field as well asthe direction of motion
It has been shown that images can be represented bytheir global phase alone [11] Here we provide a reconstruc-tion algorithm of visual scenes by only using local phaseinformation thereby demonstrating the spectrum of therepresentational capability of phase information
The Fourier shift property clearly suggests the relation-ship between the global shift of an image and the global phaseshift in the frequency domain We elevate this relationshipby computing the change of local phase to indicate motionthat appears locally in the visual scene The local phases arecomputed using window functions that tile the visual fieldwith overlapping segments making it amenable for a highlyparallel implementation In addition we propose a Radon-transform-based motion detection index on the change of
local phases for the readout of the relation between localphases and motion
Interestingly phase information has been largely ignoredin the field of linear signal processing and for good reasonPhase-based processing is intrinsically non-linear Recentresearches however showed that phase information canbe smartly employed in speech processing [12] and visualprocessing [13] For example spatial phase in an image isindicative of local features such as edges when consideringphase congruency [14] The role of spatial phase in compu-tational and biological vision emergence of visual illusionsandpattern recognition is discussed in [15] Togetherwith ourresult for motion detection these studies suggest that phaseinformation has a great potential for achieving efficient visualsignal processing
This paper is organized as follows In Section 2 weshow that local phase information can be used to faithfullyrepresent visual information in the absence of amplitudeinformation In Section 3 we develop a simple local phase-based algorithm to detectmotion in visual scenes and providean efficient way for its implementation We then provideexamples and applications of the proposed motion detectionalgorithm to motion segmentation We also compare ourresults to those obtained using motion detection algorithmsbased on optic flow techniques in Section 4 Finally wesummarize our results in Section 5
2 Representation of Visual Scenes UsingPhase Information
Theuse of complex valued transforms is widespread for bothrepresenting and processing images When represented inpolar coordinates the output of a complex valued transformof a signal can be split into amplitude and phase In thissection we define two types of phases of an image globalphase and local phase We then argue that both types ofphases can faithfully represent an image and we provide areconstruction algorithm that recovers the image from localphase information This indicates that phase informationalone can largely represent an image or video signal
It will become clear in the sections that follow that theuse of phase for representing of image and video infor-mation leads to efficient ways to implement certain typesof imagevideo processing algorithms for example motiondetection algorithms
21 The Global Phase of Images Recall that the Fouriertransform of a real valued image 119906 = 119906(119909 119910) (119909 119910) isin R2is given by
(120596119909 120596119910) = int
R2119906 (119909 119910) 119890
minus119895(120596119909119909+120596119910119910)119889119909 119889119910 (1)
with (120596119909 120596119910) isin C and (120596
119909 120596119910) isin R2
In polar coordinates the Fourier transform of 119906 can beexpressed as
(120596119909 120596119910) = (120596
119909 120596119910) 119890119895(120596119909120596119910) (2)
Computational Intelligence and Neuroscience 3
where (120596119909 120596119910) isin R is the amplitude and
120601(120596119909 120596119910) is the
phase of the Fourier transform of 119906
Definition 1 The amplitude of the Fourier transform of animage 119906 = 119906(119909 119910) (119909 119910) isin R2 is called the global amplitudeof 119906 The phase of the Fourier transform of an image 119906 =
119906(119909 119910) (119909 119910) isin R2 is called the global phase of 119906
It is known that the global phase of an image plays animportant role in the representation of natural images [11] Aclassic example is to take two images and to exchange theirglobal phases before their reconstruction using the inverseFourier transform The resulting images are slightly smearedbut largely reflect the information contained in the globalphase
22The Local Phase of Images The global phase indicates theoffset of sinusoids of different frequencies contained in theentire image However it is not intuitive to relate the globalphase to local image features such as edges and their positionin an image To study these local features it is necessary tomodify the Fourier transform such that it reflects propertiesof a restricted region of an image It is natural to consider theShort-Time (-Space) Fourier Transform (STFT)
119880(120596119909 120596119910 1199090 1199100) = int
R2119906 (119909 119910)119908 (119909 minus 119909
0 119910 minus 119910
0)
sdot 119890minus119895(120596119909(119909minus1199090)+120596119910(119910minus1199100))119889119909 119889119910
(3)
where 119908 = 119908(119909 119910) (119909 119910) isin R2 is a real valued windowfunction centered at (119909
0 1199100) isin R2 Typical choices of window
functions include the Hann window and the Gaussian win-dow The (effectively) finite support of the window restrictsthe Fourier transform to local image analysis
Similarly to the Fourier transform the STFT can beexpressed in polar coordinates as
119880(120596119909 120596119910 1199090 1199100) = 119860 (120596
119909 120596119910 1199090 1199100) 119890119895120601(12059611990912059611991011990901199100) (4)
where 119860(120596119909 120596119910 1199090 1199100) isin R is the amplitude and 120601(120596
119909 120596119910
1199090 1199100) is the phase of the STFT
Definition 2 The amplitude of the STFT of an image 119906 =
119906(119909 119910) (119909 119910) isin R2 is called the local amplitude of 119906 Thephase of the STFT of an image 119906 = 119906(119909 119910) (119909 119910) isin R2 iscalled the local phase of 119906
Note that when 119908 is a Gaussian window the STFT of 119906evaluated at (120596
119909 120596119910 1199090 1199100) can be equivalently viewed as the
response of a complex-valued Gabor receptive field
ℎ (119909 119910) = 119890minus((119909minus119909
0)2+(119910minus119910
0)2)21205902
119890minus119895(120596119909(119909minus1199090)+120596119910(119910minus1199100)) (5)
to 119906In this case the window of the STFT clearly is given by
119908 (119909 119910) = 119890minus(1199092+1199102)21205902
(6)
Therefore the STFT can be realized by an ensemble of Gaborreceptive fields that are common in modeling simple andcomplex cells (neurons) in the primary visual cortex [16]
23 Reconstruction of Images from Local Phase Amplitudeand phase can be interpreted as measurementsprojectionsof images that are indicative of their information contentClassically when both the global amplitude and phase areknown it is straightforward to reconstruct the image Thereconstruction calls for computing the inverse Fourier trans-form given the global amplitude and phase Similarly whenusing local amplitude and phase if the sampling functionsform a basis or frame in a space of images the reconstructionis provided by the formalism of wavelet theory for exampleusing Gabor wavelets [17]
Amplitude or phase represents partial informationextracted from visual scenesThey are obtained via nonlinearsampling that is a nonlinear operation for extracting theamplitude andphase information from imagesThenonlinearoperation makes reconstruction from either amplitude orphase alone difficult Earlier studies and recent developmentin solving quadratic constraints however suggest that it ispossible to reconstruct images from global or local amplitudeinformation [18 19]
While computing the amplitude requires a second order(quadratic) nonlinearity computing the phase calls for higherorder nonlinear operators (Volterra kernels) It is possiblehowever to reconstruct up to a constant scale an image fromits global phase information alone without explicitly usingthe amplitude information An algorithm was provided forsolving this problem in the discrete signal processing domainin [11]The algorithm smartly avoids using the inverse tangentfunction by reformulating the phase measurements as a set oflinear equations that are easy to solve
Using a similar argument we demonstrate in the follow-ing that up to a constant scale a bandlimited signal 119906 =
119906(119909 119910) (119909 119910) isin R2 can be reconstructed from its local phasealone We first formulate the encoding of an image 119906 by localphase using the Gabor receptive fields as a special case Itis straightforward then to formulate the problem with localphase computed from other types of STFTs
Formally we consider an image 119906 = 119906(119909 119910) on thedomain R2 to be an element of a space of trigonometricpolynomialsH of the form
119906 (119909 119910) =
119871119909
sum
119897119909=minus119871119909
119871119910
sum
119897119910=minus119871119910
119888119897119909119897119910
119890119897119909119897119910
(119909 119910) (7)
where
119890119897119909119897119910
= exp(119895119897119909
Ω119909
119871119909
119909) exp(119895119897119910
Ω119910
119871119910
119910)
119897119909= minus119871119909 119871
119909 119897119910= minus119871119910 119871
119910
(8)
are the set of basis functions ofHΩ119909 119871119909are the bandwidth
and the order respectively of the space in the 119909 dimensionand Ω
119910 119871119910are the bandwidth and the order respectively of
the space in the 119910 dimensionH is a Reproducing Kernel Hilbert Space with inner
product [20 21]
⟨1199061 1199062⟩ = int
[0119879119909]times[0119879
119910]
1199061(119909 119910) 119906
2(119909 119910)119889119909 119889119910 (9)
4 Computational Intelligence and Neuroscience
where 119879119909= 2120587119871
119909Ω119909 119879119910= 2120587119871
119910Ω119910are the period of the
space in the 119909 and 119910 dimensions respectivelyConsider a bank of119873 Gabor receptive fields
ℎ119896119897119898119899
(119909 119910) = T119896119897(119908 (119909 119910) 119890
minus119895(120596119909119898119909+120596119910119899119910)) (10)
where T119896119897is the translation operator with T
119896119897119906 = 119906(119909 minus
1198961198870 119910 minus 119897119887
0) 119896 119897 isin Z 119887
0gt 0 0 le 119896119887
0le 119879119909 and 0 le
1198971198870le 119879119910 and 120596
119909119898
= 1198981205960 120596119910119899
= 1198991205960 119898 119899 isin Z 120596
0gt 0
minusΩ119909le 119898120596
0le Ω119909 and minusΩ
119910le 1198991205960le Ω119910 The responses of
the Gabor receptive fields to the input 119906(119909 119910) are given by
int
R2119906 (119909 119910) ℎ
119896119897119898119899(119909 119910) 119889119909 119889119910 = int
R2119906 (119909 119910)
sdot 119908 (119909 minus 1198961198870 119910 minus 119897119887
0) 119890minus119895(120596119909119898(119909minus119896119887
0)+120596119910119899(119910minus1198971198870))119889119909 119889119910
= 119860119896119897119898119899
119890119895120601119896119897119898119899
(11)
where 119860119896119897119898119899
ge 0 119860119896119897119898119899
isin R is the local amplitude and120601119896119897119898119899
isin [0 2120587) is the local phaseDividing both sides of (11) by 119890119895120601119896119897119898119899 we have
int
R2119906 (119909 119910)119908 (119909 minus 119896119887
0 119910 minus 119897119887
0)
sdot 119890minus119895(120596119909119898(119909minus119896119887
0)+120596119910119899(119910minus1198971198870)+120601119896119897119898119899)119889119909 119889119910 = 119860
119896119897119898119899
(12)
Since 119860119896119897119898119899
isin R we have
int
R2119906 (119909 119910)119908 (119909 minus 119896119887
0 119910 minus 119897119887
0)
sdot cos (120596119909119898
(119909 minus 1198961198870) + 120596119910119899
(119910 minus 1198971198870) + 120601119896119897119898119899
) 119889119909 119889119910
= 119860119896119897119898119899
(13)
int
R2119906 (119909 119910)119908 (119909 minus 119896119887
0 119910 minus 119897119887
0)
sdot sin (120596119909119898
(119909 minus 1198961198870) + 120596119910119899
(119910 minus 1198971198870) + 120601119896119897119898119899
) 119889119909 119889119910
= 0
(14)
Note that119908(119909 minus 1198961198870 119910 minus 119897119887
0) sin(120596
119909119898
(119909 minus 1198961198870) + 120596119910119899
(119910 minus 1198971198870) +
120601119896119897119898119899
) is a real-valued Gabor receptive field with a preferredphase at 120601
119896119897119898119899+ 1205872 minus 120596
119909119898
1198961198870minus 120596119910119899
1198971198870
Remark 3 Assuming that the local phase information 120601119896119897119898119899
is obtained via measurements that is filtering the image 119906
with pairs of Gabor receptive fields (10) the set of linearequations (14) has a simple interpretation the image 119906 isorthogonal to the space spanned by the functions
119908 (119909 minus 1198961198870 119910 minus 119897119887
0)
sdot sin (120596119909119898
(119909 minus 1198961198870) + 120596119910119899
(119910 minus 1198971198870) + 120601119896119897119898119899
)
(15)
where (119896 119897 119898 119899) isin I with I = (119896 119897 119898 119899) isin Z4 | 0 le 1198961198870le
119879119909 0 le 119897119887
0le 119879119910 minusΩ119909le 119898120596
0le Ω119909 minusΩ119910le 1198991205960le Ω119910
We are now in the position to provide a reconstructionalgorithm of the image from phase 120601
119896119897119898119899 (119896 119897 119898 119899) isin I
Lemma 4 119906 can be reconstructed from 120601119896119897119898119899
(119896 119897 119898 119899) isin Ias
119906 (119909 119910) =
119871119909
sum
119897119909=minus119871119909
119871119910
sum
119897119910=minus119871119910
119888119897119909119897119910
119890119897119909119897119910
(119909 119910) (16)
with
Φ119888 = 0 (17)
whereΦ is a matrix whose 119901th row and 119902th column entry are
[Φ]119901119902
= int
R2119908 (119909 minus 119896119887
0 119910 minus 119897119887
0)
sdot sin (120596119909119898
(119909 minus 1198961198870) + 120596119910119899
(119910 minus 1198971198870) + 120601119896119897119898119899
)
sdot 119890119897119909119897119910
(119909 119910) 119889119909 119889119910
(18)
Here 119901 traverses the set I and 119902 = (2119871119910+ 1)(119897119909+ 119871119909) + (119897119910+
119871119910+ 1) 119888 is a vector of the form
119888 = [119888minus119871119909minus119871119910
119888minus119871119909minus119871119910+1 119888
minus119871119909119871119910
119888minus119871119909+1minus119871
119910
119888minus119871119909+1minus119871
119910+1 119888
minus119871119909+1119871119910
119888119871119909minus119871119910
119888119871119909minus119871119910+1
119888119871119909119871119910
]
119879
(19)
that belongs to the null space of Φ A necessary conditionfor perfect reconstruction of 119906 up to a constant scale is that119873 ge (2119871
119909+ 1)(2119871
119910+ 1) minus 1 where 119873 is the number of phase
measurements
Proof Substituting (7) into (14) we obtain119871119909
sum
119897119909=minus119871119909
119871119910
sum
119897119910=minus119871119910
119888119897119909119897119910
int
R2119908 (119909 minus 119896119887
0 119910 minus 119897119887
0)
sdot sin (120596119909119898
(119909 minus 1198961198870) + 120596119910119899
(119910 minus 1198971198870) + 120601119896119897119898119899
)
sdot 119890119897119909119897119910
(119909 119910) 119889119909 119889119910 = 0
(20)
for all 119896 119897 119898 119899 isin Z Therefore we have
Φ119888 = 0 (21)
inferring that 119888 is in the null space ofΦIf 119873 lt (2119871
119909+ 1)(2119871
119910+ 1) minus 1 it follows from the rank-
nullity theorem that dim(null(Φ)) gt 1 leading to multiplelinearly independent solutions toΦ119888 = 0
Example 5 In Figure 2 an example of reconstruction of animage is shown using only local phase information Thereconstructed signal was scaled to match the original signalThe SNR of the reconstruction is 4448 [dB] An alternativeway to obtain a unique reconstruction is to include anadditional measurement for example the mean value of thesignal int
R2119906(119909 119910)119889119909 119889119910 to the system of linear equations (14)
Computational Intelligence and Neuroscience 5
(a) (b) (c)
Figure 2 An example of reconstruction of image from local phase information (a) Original image the dimension of the space is 16129(b) Reconstructed image from 20102 phase measurements scaled to match the original (c) Error
3 Visual Motion Detection fromPhase Information
In this section we consider visual fields that change as afunction of time For notational simplicity 119906 will denote herethe space-time intensity of the visual field
31 The Global Phase Equation for Translational Motion Let119906 = 119906(119909 119910 119905) (119909 119910) isin R2 119905 isin R be a visual stimulus If thevisual stimulus is a pure translation of the signal at 119906(119909 119910 0)that is
119906 (119909 119910 119905) = 119906 (119909 minus 119904119909(119905) 119910 minus 119904
119910(119905) 0) (22)
where
119904119909(119905) = int
119905
0
V119909(119904) 119889119904
119904119910(119905) = int
119905
0
V119910(119904) 119889119904
(23)
are the total length of translation at time 119905 in each dimen-sion and V
119909(119905) and V
119910(119905) are the corresponding instanta-
neous velocity components then the only difference between119906(119909 119910 119905) and 119906(119909 119910 0) in the Fourier domain is captured bytheir global phase More formally consider the following
Lemma 6 The change (derivative) of the global phase is givenby
119889120601 (120596119909 120596119910 119905)
119889119905
= minus120596119909V119909(119905) minus 120596
119910V119910(119905)
= minus [120596119909 120596119910] [V119909(119905) V119910(119905)]
119879
(24)
where by abuse of notation 120601(120596119909 120596119910 119905) denotes the global
phase of 119906(119909 119910 119905) and 120601(120596119909 120596119910 0) is the initial condition
Proof If (F119906(sdot sdot 0)) (120596119909 120596119910) is the 2D (spatial) Fourier
transform of 119906 = 119906(119909 119910 0) (119909 119910) isin R2 by the Fourier shifttheorem we have
(F119906 (sdot sdot 119905)) (120596119909 120596119910)
= (F119906 (sdot sdot 0)) (120596119909 120596119910) 119890minus119895(120596119909119904119909(119905)+120596
119910119904119910(119905))
(25)
For a certain frequency component (120596119909 120596119910) the change of its
global phase over time amounts to
119889120601 (120596119909 120596119910 119905)
119889119905
= minus120596119909
119889119904119909(119905)
119889119905
minus 120596119910
119889119904119910(119905)
119889119905
= minus120596119909V119909(119905) minus 120596
119910V119910(119905)
= minus [120596119909 120596119910] [V119909(119905) V119910(119905)]
119879
(26)
Therefore in the simple case where the entire visual field isshifting the derivative of the phase of Fourier componentsindicates motion and it can be obtained by the innerproduct between the component frequency and the velocityvector
32 The Change of Local Phase
321The Local Phase Equation for Translational Motion Theanalysis in Section 31 applies to global motion This type ofmotion occurs most frequently when the imaging deviceeither an eye or a cameramoves Visualmotion in the naturalenvironment however ismore diverse across the screen sinceit is often time produced by multiple moving objects Theobjects can be small and the motion more localized in thevisual field
Taking the global phase of 119906 will not simply revealwhere motion of independent objects takes place or theirdirectionvelocity of motion The ease of interpretation ofmotion by using the Fourier transform however motivates
6 Computational Intelligence and Neuroscience
us to reuse the same concept in detecting local motion Thiscan be achieved by restricting the domain of the visual fieldwhere the Fourier transform is applied
To be able to detect local motion we consider thelocal phase of 119906(119909 119910 119905) by taking the STFT with windowfunction 119908(119909 119910) Note that the STFT and its ubiquitousimplementation in DSP chips can be extensively used in anydimension For simplicity and without loss of generality weconsider the window to be centered at (0 0) The STFT isgiven by
int
R2119906 (119909 119910 119905) 119908 (119909 119910) 119890
minus119895(120596119909119909+120596119910119910)119889119909 119889119910
= 11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)
(27)
where by abuse of notation 11986000(120596119909 120596119910 119905) is the amplitude
and 12060100(120596119909 120596119910 119905) the local phase
Before we move on to the mathematical analysis we canintuitively explain the relation between the change in localphase and visual motion taking place across the windowsupport First if the stimulus undergoes a uniform changeof intensity or it changes proportionally over time due tolighting conditions for example the local phase does notchange since the phase is invariant with respect to intensityscaling Therefore the local phase does not change for suchnonmotion stimuli Second a rigid edge moving across thewindow support will induce a phase change
For a strictly translational signal within the windowsupport (footprint) for example
119906 (119909 119910 119905) = 119906 (119909 minus 119904119909(119905) 119910 minus 119904
119910(119905) 0)
for (119909 119910) isin supp (119908 (119909 119910))
(28)
where 119904119909(119905) and 119904
119910(119905) are as defined in (23) we have the
following result
Lemma 7 Consider the following
11988912060100
119889119905
(120596119909 120596119910 119905) = minus
119889119904119909(119905)
119889119905
120596119909minus
119889119904119910(119905)
119889119905
120596119910
+ v00(120596119909 120596119910 119905)
(29)
where by abuse of notation 12060100(120596119909 120596119910 119905) is the local phase
of 119906(119909 119910 119905) and 12060100(120596119909 120596119910 0) is the initial condition The
functional form of the term v00
= v00(120596119909 120596119910 119905) is provided
in the Appendix
Proof The derivation of (29) and the functional form of v00
are given in the Appendix
Remark 8 Wenotice that the derivative of the local phase hassimilar structure to that of the global phase but for the addedterm v
00Through simulations we observed that the first two
terms in (29) dominate over the last term for an ON or OFFmoving edge [3] For example Figure 3 shows the derivativeof the local phase given in (29) for an ON edge moving withvelocity (40 0) pixelssec
42
0minus2
minus4
100
d120601dt
120596yminus250minus200minus150minus100minus50
050
150
200250
31 2minus4 0minus1minus3 minus2 4
120596x
Figure 3 Derivative of the local phase for an ON edge moving inthe positive 119909 direction
Remark 9 Note that 12060100(120596119909 120596119910 119905) may not be differen-
tiable even if 119906(119909 119910 119905) is differentiable particularly when11986000(120596119909 120596119910 119905) = 0 For example the spatial phase can jump
from a positive value to zero when11986000(120596119909 120596119910 119905) diminishes
This also suggests that the instantaneous local spatial phaseis less informative about a region of a visual scene whenever11986000(120596119909 120596119910 119905) is close to zero Nevertheless the time deriva-
tive of the local phase can be approximated by applying ahigh-pass filter to 120601
00(120596119909 120596119910 119905)
322 The Block Structure for Computing the Local Phase Weconstruct Gaussian windows along the 119909 119910 dimensions TheGaussian windows are defined as
(T119896119897119908) (119909 119910) = 119890
minus((119909minus119909119896)2+(119909minus119910
119897)2)21205902
(30)
where 119909119896= 1198961198870 119910119897= 1198971198870 in which 119887
0isin Z+ is the distance
between two neighboring windows and 1 le 1198961198870le 119875119909 1 le
1198971198870le 119875119910 where 119875
119909 119875119910isin Z+ are the number of pixels of the
screen in 119909 and 119910 directions respectivelyWe then take the 2D Fourier transform of the windowed
video signal 119906(119909 119910 119905)(T119896119897119908)(119909 119910) and write in polar form
int
R2119906 (119909 119910 119905) (T
119896119897119908) (119909 119910) 119890
minus119895(120596119909(119909minus119909119896)+120596119910(119910minus119910119897))119889119909 119889119910
= 119860119896119897(120596119909 120596119910 119905) 119890119895120601119896119897(120596119909120596119910119905)
(31)
The above integral can be very efficiently evaluated usingthe 2D FFT in discrete domain defined on 119872 times 119872 blocksapproximating the footprint of the Gaussian windows Forexample the standard deviation of the Gaussian windows weuse in the examples in Section 4 is 4 pixels A block of 32times32
pixels (119872 = 32) is sufficient to cover the effective support(or footprint) of the Gaussian window At the same time thesize of the block is a power of 2 which is most suitable forFFT-based computationThe processing of each block (119896 119897) isindependent of all other blocks thereby parallelism is readilyachieved
Note that the size of the window is informed by thesize of the objects one is interested in locating Measure-ments of the local phase using smaller window functions
Computational Intelligence and Neuroscience 7
60
50
40
30
20
10
y (pi
xel)
01
02
03
04
05
06
07
08
09
1
20 30 40 50 6010x (pixel)
(a)
0
02
04
06
08
1
Mag
nitu
de (a
u)
10 20 30 40 50 60 700y (pixel)
(b)
Figure 4 Example of block structure (a) Four yellow disks show 4 Gaussian window functions with translation parameter 119896 = 3 119897 = 3
(top-left) 119896 = 3 119897 = 7 (bottom-left) 119896 = 7 119897 = 3 (top-right) and 119896 = 7 119897 = 7 (bottom-right) The red solid square shows a 32 times 32-pixel blockapproximating the Gaussian window with 119896 = 3 119897 = 3 and the green dashed square shows a 32 times 32-pixel block approximating the Gaussianwindow with 119896 = 3 119897 = 7 (b) Cross section of all 11 Gaussian windows with translation parameters 119896 = 7 119897 isin [0 10] The cross section istaken as indicated by the magenta line in (a) and the red and blue curve correspond to cross sections of the two Gaussian windows shown in(a) centered on the magenta line
are less robust to noise Larger windows would enhanceobject motion detection if the object size is comparable tothe window size However there would be an increasedlikelihood of independent movement of multiple objectswithin the same window which is not modeled here andthereby may not be robustly detected
Therefore for each block (119896 119897) we obtain 1198722 measure-
ments of the phase 120601119896119897(120596119909119898
120596119910119898
119905) at every time instant 119905with (120596
119909119898
120596119910119898
) isin D2 where
D2= (120596
119909119898
= 1198981205960 120596119910119899
= 1198991205960) 119898 119899 = minus1198722
minus 1198722 + 1 1198722 minus 1
(32)
with 1205960= 2120587119872 We then compute the temporal derivative
of the phase that is (119889120601119896119897119889119905)(120596
119909 120596119910 119905) for (120596
119909 120596119910) isin D2
We further illustrate an example of the block structurein Figure 4 Figure 4(a) shows an example of an image of64 times 64 pixels Four Gaussian windows are shown each witha standard deviation of 4 pixels The distance between thecenters of two neighboring Gaussian windows is 6 pixelsThe red solid square shows a 32 times 32-pixel block with 119896 =
3 119897 = 3 which encloses effective support of the Gaussianwindow on top-left (119896 = 0 119897 = 0 is the block with Gaussianwindow centered at pixel (1 1)) The green dashed squareshows another 32 times 32-pixel block with 119896 = 3 119897 = 7 Thetwo Gaussian windows on the right are associated with theblocks 119896 = 7 119897 = 3 and 119896 = 7 119897 = 7 respectively Crosssection of all Gaussian windows with 119896 = 7 119897 isin [0 10] that isthose centered on themagenta line are shown in Figure 4(b)The red and blue curve in Figure 4(b) correspond to the two
Gaussian windows shown in Figure 4(a) Figure 4(b) alsosuggests that some of the Gaussian windows are cut off on theboundaries This is however equivalent to assuming that thepixel values outside the boundary are always zero and it willnot significantly affect motion detection based on the changeof local phase
Since the phase and thereby the phase change is noisierwhen the local amplitude is low an additional denoisingstep can be employed to discount the measurements of(119889120601119896119897119889119905)(120596
119909 120596119910 119905) for low amplitude values 119860
119896119897(120596119909 120596119910 119905)
The denoising is given by
119889120601119896119897
119889119905
(120596119909 120596119910 119905)
sdot
119860119896119897(120596119909 120596119910 119905)
(11198722)sum(120596119909120596119910)isinD2 119860119896119897 (120596119909 120596119910 119905) + 120598
(33)
where 120598 gt 0 is a constant and (120596119909 120596119910) isin D2
33 The Phase-Based Detector We propose here a block FFTbased algorithm to detect motion using phase informationSuch an algorithm is due to its simplicity and parallelismhighly suitable for an in silico implementation
331 Radon Transform on the Change of Phases We exploitthe approximately linear structure of the phase derivative forblocks exhibiting motion by computing the Radon transformof (119889120601
119896119897119889119905)(120596
119909 120596119910 119905) over a circular bounded domain 119862 =
(120596119909 120596119910) | (120596119909 120596119910) isin D2 1205962
119909+ 1205962
119910lt 1205872
8 Computational Intelligence and Neuroscience
The Radon transform of the change of phase in thedomain 119862 is given by
(R119889120601119896119897
119889119905
) (120588 120579 119905) = int
R
119889120601119896119897
119889119905
(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588
sdot sin 120579 + 119904 sdot cos 120579 119905) 1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579
+ 119904 sdot cos 120579) 119889119904
(34)
where
1119862(120596119909 120596119910) =
1 if (120596119909 120596119910) isin 119862
0 otherwise(35)
The Radon transform (R(119889120601119896119897119889119905))(120588 120579 119905) evaluated at
a particular point (1205880 1205790 1199050) is essentially an integral
of (119889120601119896119897119889119905)(120596
119909 120596119910 1199050) along a line oriented at angle
1205872 + 1205790with the 120596
119909axis and at distance |120588
0| along the
(cos(1205790) sin(120579
0)) direction from (0 0)
If for a particular 119896 and 119897 we have (119889120601119896119897119889119905)(120596
119909 120596119910 119905) =
minusV119909(119905)120596119909minus V119910(119905)120596119910 we have
(R (119889120601119896119897119889119905)) (120588 120579 119905)
c (120588 120579)
= 120588 [minusV119909(119905) cos 120579 minus V
119910(119905) sin 120579]
(36)
where
c (120588 120579)
= int
R
1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579 + 119904 sdot cos 120579) 119889119904
(37)
c(120588 120579) is a correction term due to different length of lineintegrals for different values of (120588 120579) in the bounded domain119862
After computing the Radon transform of (119889120601119896119897119889119905)(120596
119909
120596119910 119905) for every block (119896 119897) at time 119905
0 we compute the Phase
Motion Indicator (PMI) defined as
PMI119896119897= max120579isin[0120587)
sum
120588
100381610038161003816100381610038161003816100381610038161003816
(R (119889120601119896119897119889119905)) (120588 120579 119905
0)
c (120588 120579)
100381610038161003816100381610038161003816100381610038161003816
(38)
If the PMI119896119897
is larger than a chosen threshold motion isdeemed to occur in block (119896 119897) at time 119905
0
Using the Radon transform makes it easier to separaterigid motion from noise Since the phase is quite sensitiveto noise particularly when the amplitude is very smallthe change of phase under noise may have comparablemagnitude to that due to motion as mentioned earlier Thechange of phase under noise however does not possess thestructure suggested by (29) in the (120596
119909 120596119910) domain Instead
it appears to be more randomly distributed Consequently
the PMI value is comparatively small for these blocks (see alsoSection 333)
Moreover the direction of motion for block (119896 119897) wheremotion is detected can be easily computed as
120579119896119897
= 120587(
sign (sum120588gt0
(R (119889120601119896119897119889119905)) (120588 120572
119896119897 1199050) c (120588 120572
119896119897)) + 1
2
)
+ 120572119896119897
(39)
where
120572119896119897= argmax120579isin[0120587)
sum
120588
100381610038161003816100381610038161003816100381610038161003816
(R (119889120601119896119897119889119905)) (120588 120579 119905
0)
c (120588 120579)
100381610038161003816100381610038161003816100381610038161003816
(40)
This follows from (36)
332 The Phase-Based Motion Detection Algorithm We for-mally summarize the above analysis as Algorithm 1 Figure 5shows a schematic diagram of the proposed phase-basedmotion detection algorithm
The algorithm is subdivided into two parts The first partcomputes local phase changes and the second part is thephase-based motion detector
In the first part the screen is divided into overlappingblocks For example the red green and blue blocks in theplane ldquodivide into overlapping blocksrdquo correspond to thesquares of the same color covering the video stream AGaussian window is then applied on each block followed bya 2D FFT operation that is used to extract the local phase Atemporal high-pass filter is then employed to extract phasechanges
In the second part the PMI is evaluated for each blockbased on the Radon transform of the local phase changes ineach block Motion is detected for blocks with PMI largerthan a preset threshold and the direction of motion iscomputed as in (39)
It is easy to notice that the algorithm can be highlyparallelized
333 Example We provide an illustrative example inFigure 6 showing how motion is detected using Algorithm 1The full video of this example can be found in SupplementaryVideo S1 see Supplementary Material available online athttpdxdoiorg10115520167915245 Figure 6(a) depicts astill from the ldquohighway videordquo in the Change Detection 2014dataset [22] evaluated at a particular time 119905
0 As suggested
by the algorithm the screen in Figure 6(a) is divided into26 times 19 overlapping blocks and the window functions areapplied to each block Local phases can then be extractedfrom the 2DFFT of eachwindowed block and the local phasechanges are obtained by temporal high-pass filtering Thephase change is shown in Figure 6(b) for all blocks with block(12 11) enlarged in Figure 6(c) and block (23 6) enlarged inFigure 6(d) (see also the plane ldquo2D FFT and extract phasechangerdquo in Figure 5) Note that at the time of the videoframe block (12 11) covers a part of the vehicle in motion
Computational Intelligence and Neuroscience 9
Input Visual stream or Video data 119906(119909 119910 119905)Output Detected MotionConstruct Gaussian window of size119872times119872 denote as 119908for 119905 = Δ119879 2Δ119879 119873Δ119879 do
Divide the screen of frame at time 119905 into multiple119872times119872 blocks 119906119896119897(119905) with overlaps
foreach block 119906119896119897(119905) do
Multiply pixel-wise the block 119906119896119897(119905) with the Gaussian window 119906
119896119897(119905)119908
Take119872times119872 2D FFT of 119906119896119897(119905)119908 denote phase as 120601
119896119897(120596119909 120596119910 119905) and amplitude as 119860
119896119897(120596119909 120596119910 119905)
foreach frequency (120596119909 120596119910) do
Highpass filter temporally 120601119896119897(120596119909 120596119910 119905) denote 1206011015840
119896119897(120596119909 120596119910 119905)
Optionally denoise by 1206011015840119896119897(120596119909 120596119910 119905) larr 120601
1015840
119896119897(120596119909 120596119910 119905)(119860
119896119897(120596119909 120596119910 119905)((1119872
2) sum(120596119909 120596119910)isinD
2 119860119896119897(120596119909 120596119910 119905) + 120598))
endCompute the radon transform (R120601
1015840
119896119897)(120588 120579 119905)
Compute PMI119896119897(119905) according to (38)
if PMI119896119897(119905) gt threshold then
Motion Detected at block (119896 119897) at time 119905Compute direction of motion according to (39)
endend
end
Algorithm 1 Phase-based motion detection algorithm using the FFT
y
120596x
120596y
y
t
120596x
120596y
x
y
x
y
x
x
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
Divided intooverlapping blocks
Apply window function
2D FFT andextract phase change
Radon transformand extract strength
and orientation of plane
Thresholding
Computephase
change
The phase-baseddetector
Figure 5 Schematic diagram of proposed phase-based motion detection algorithm
10 Computational Intelligence and Neuroscience
in the front and block (23 6) corresponds to an area of thehighway pavement where no motion occurs
Figure 6(f) depicts for each block (119896 119897) the maximumphase change over all (120596
119909 120596119910) isin D2 that is
max(120596119909120596119910)isinD2
10038161003816100381610038161003816100381610038161003816
119889120601119896119897
119889119905
(120596119909 120596119910 1199050)
10038161003816100381610038161003816100381610038161003816
(41)
We observe from the figure that for regions with lowamplitude such as the region depicting the road whenthe normalization constant is absent the derivative of thephase can be noisy For these blocks the maximum of|(119889120601119896119897119889119905)(120596
119909 120596119910 1199050)| over all (120596
119909 120596119910) isin D2 is comparable
to the maximum obtained for blocks that cover the vehiclesin motion
However (29) suggests that the local phase change frommultiple filter pairs centered at the same spatial position(119896 119897) can provide a constraint to robustly estimate motionand its direction Given the block structure employed in thecomputation of the local phase it is natural to utilize phasechange information from multiple sources
Indeed if for a particular block (119896 119897) (119889120601119896119897119889119905)(120596
119909
120596119910 119905) = minusV
119909(119905)120596119909minus V119910(119905)120596119910 then it is easy to see that (119889120601
119896119897
119889119905)(120596119909 120596119910 119905) will be zero on the line V
119909(119905)120596119909+ V119910(119905)120596119910= 0
and have opposite sign on either side of this line For examplein Figures 6(b) and 6(c) (119889120601
119896119897119889119905)(120596
119909 120596119910 1199050) clearly exhibits
this property for blocks that cover a vehicle in motion ThePMI is a tool to evaluate this property
Finally the PMIs for all blocks are shown compactlyin a heat map in Figure 6(e) The figure shows clearly thatthe blocks corresponding to the two moving vehicles have ahigh PMI value while the stationary background areas havea low PMI value allowing one to easily detect motion byemploying simple thresholding (see also the plane ldquoRadontransform and extract strength orientation of planerdquo inFigure 5) In addition the orientation ofmotion in each blockis readily observable even by inspection in Figure 6(b) by aline separating the yellow part and blue part in each blockFurther results about the direction of motion are presentedin Section 4
34 Relationship to Biological Motion Detectors A straight-forward way to implement local motion detectors is to applya complex-valued Gabor receptive field (5) to the video signal119906 and then take the derivative of the phase with respect totime or apply a high-pass filter on the phase to approximatethe derivative
We present here an alternate implementation withoutexplicitly computing the phaseThiswill elucidate the relationbetween the phase-based motion detector presented in theprevious section and some elementary motion detectionmodels used in biology such as theReichardtmotion detector[7] and motion energy detector [9]
Assuming that the local phase 12060100(120596119909 120596119910 119905) is differen-
tiable we have (see also the Appendix)
11988912060100(120596119909 120596119910 119905)
119889119905
=
(119889119887 (120596119909 120596119910 119905) 119889119905) 119886 (120596
119909 120596119910 119905) minus (119889119886 (120596
119909 120596119910 119905) 119889119905) 119887 (120596
119909 120596119910 119905)
[119886 (120596119909 120596119910 119905)]
2
+ [119887 (120596119909 120596119910 119905)]
2 (42)
where 119886(120596119909 120596119910 119905) and 119887(120596
119909 120596119910 119905) are respectively the real
and imaginary parts of 11986000(120596119909 120596119910 119905)11989011989512060100(120596119909120596119910119905)
We notice that the denominator of (42) is the square ofthe local amplitude of 119906 and the numerator is of the formof a second order Volterra kernel This suggests that thetime derivative of the local phase can be viewed as a secondorder Volterra kernel that processes two normalized spatiallyfiltered inputs V
1and V2
We consider an elaborated Reichardt motion detector asshown in Figure 7 It is equipped with a quadrature pair ofGabor filters whose outputs are 119903
1(119905) = 119886(120596
119909 120596119910 119905) and
1199032(119905) = 119887(120596
119909 120596119910 119905) respectively for a particular value of
(120596119909 120596119910) The pair of Gabor filters that provide these outputs
are the real and imaginary parts of119908(119909 119910)119890minus119895(120596119909119909+120596119910119910) It also
consists of a temporal high-pass 1198921(119905) and temporal low-
pass filter 1198922(119905) [9] The output of the elaborated Reichardt
detector follows the diagram in Figure 7 and can be expressedas
(1199032lowast 1198921) (119905) (119903
1lowast 1198922) (119905) minus (119903
1lowast 1198921) (119905) (119903
2lowast 1198922) (119905) (43)
The response can also be characterized by a second orderVolterra kernel We notice the striking similarity between
(43) and the numerator of (42) In fact the phase-basedmotion detector shares some properties with the Reichardtmotion detector For example it is straightforward to see thata single phase-basedmotion detector is tuned to the temporalfrequency of a moving sinusoidal grating
Since the motion energy detector is formally equivalentto an elaborated Reichardt motion detector [9] the structureof the motion energy detector with divisive normalization isalso similar to the phase-based motion detector
4 Exploratory Results
In this section we apply the phase-based motion detectionalgorithm on several video sequences and demonstrate itsefficiency and effectiveness in detecting local motion Themotion detection algorithm is compared to two well-knownbiological motion detectors namely the Reichardt motiondetector and the Barlow-Levick motion detector We thenshow that the detected local motion can be used in motionsegmentation tasks The effectiveness of the segmentationis compared to segmentation using motion informationobtained from a widely used optic flow based algorithmavailable in the literature [4]
Computational Intelligence and Neuroscience 11
(f) (e)
(b)
(a)
(d)
(c)
minus120587 minus1205872 0 1205872
10080604020
y (pi
xel)
100 15050x (pixel)
0020406081
15
10
5
l
01234567+
15
10
5
l
0510152025+
minus6
minus4
minus2
0
2
4
6
181716151413121110
9876543210
l
2 4 6 8 10 12 14 16 18 20 22 240k
minus6minus4minus20246
minus6minus4minus20246
120596x
minus120587 minus1205872 0 1205872
120596x
1205872
0
minus1205872
minus120587
120596y
1205872
0
minus1205872
minus120587
120596y
10 15 20 255k
10 20 255 15k
Figure 6 Evaluating the motion detection algorithm at time 1199050(see also Supplementary Video S1) (a) A still from a 156 times 112-pixel video at
time 1199050 (b) (119889120601
119896119897119889119905)(120596
119909 120596119910 1199050) for all 26times19 blocks each of size 32times32The blocks are concatenated to create a large ldquophase imagerdquo with their
relative neighbors kept (c) (1198891206011211
119889119905)(120596119909 120596119910 1199050) shown in the (120596
119909 120596119910) isin D2 space (d) (119889120601
236119889119905)(120596
119909 120596119910 1199050) shown in the (120596
119909 120596119910) isin D2
space (e) Phase Motion Indicator (PMI) for all blocks at 1199050 Each pixel (119896 119897) is color coded for PMI
119896119897 (f) max
(120596119909 120596119910)isinD2 |(119889120601119896119897119889119905)(120596
119909 120596119910 1199050)|
for all blocks at 1199050 Each pixel (119896 119897) is color coded for this maximum value of block (119896 119897)
g1(t)g2(t) g1(t) g2(t)
times times
minus++
SpatialGaborfilter
SpatialGaborfilter
r1(t) r2(t)
u(x y t)
Figure 7 Diagram of an elaborated Reichardt motion detector Itconsists of a quadrature pair of Gabor filters whose outputs 119903
1and 1199032
provide inputs to the high-pass filters 1198921(119905) and the low-pass filters
1198922(119905)
Table 1 Processing Speeds of the proposed algorithm the Reichardtmotion detector and the Barlow-Levick Motion detector
Screen size Processing capability (frames per second)Proposed Reichardt Barlow-Levick
320 times 240 420 790 800720 times 576 135 685 6901280 times 720 60 274 2931920 times 1080 27 191 194
41 Efficient Parallel Implementation The algorithm wasimplemented in PyCUDA [23] and tested on an NVIDIAGeForce GTX TITAN GPU All computations use singleprecision floating points The processing speeds of the algo-rithm for several screen sizes are listed in Table 1 Clearlythe proposed phase-based motion detection algorithm hasreal-time capability to process video even with full HighDefinition screen size
For comparison we implemented the Reichardt motiondetector and the Barlow-Levick motion detector Theirrespective diagrams are shown in Figure 8 Note that we
12 Computational Intelligence and Neuroscience
ukl(t) uk+1l(t)
g1(t)
g2(t)
g1(t)
g2(t)
+ minus+
timestimes
(a)
minus
minus minus
+
ukl(t) uk+1l(t)
g1(t)g1(t)
g2(t) g2(t)
++
+
(b)
Figure 8 (a) A Reichardt motion detector detecting motion between the two points 119906119896119897and 119906
119896+1119897in the blurred and downsampled video
(b) A Barlow-Levick motion detector detecting motion between the two points 119906119896119897and 119906
119896+1119897in the blurred and downsampled video 119892
1(119905)
denotes a high-pass filter and 1198922(119905) denotes a low-pass filter
moved the high-pass filter ℎ1(119905) to the front of the low-pass fil-
ter ℎ2(119905) This configuration provides a superior performance
to the one in Figure 7 For both the Reichardt and the Barlow-Levick detectors the videos are first blurred by a Gaussianfilter (with the same variance as the Gaussian window in (30)used for the phase-based motion detector) and subsampledat the center of each overlapping block in the phase-basedmotion detector The subsampled video then provides inputsto two 2D arrays of the circuits shown in Figure 8 one forthe horizontal direction and one for the vertical directionThe outputs of the horizontal and vertical motion circuitsform an array of motion vectors that indicate the strengthand direction of motion The three tested motion detectorshave the same number of outputs as a result The processingspeeds of the Reichardt motion detector and the Barlow-Levick motion detector both implemented in PyCUDA andtested on the same GPU are shown in Table 1
Note that the Reichardt motion detector and the Barlow-Levick motion detector are highly efficient due to the sim-plicity of their algorithmsThe phase-basedmotion detectionalgorithm however is a much more sophisticated algorithmand yet it can be implemented in real-time using parallelcomputing devices
The fast GPU implementation is based on the FFTand Matrix-Matrix multiplication It is expected that thoseoperations can be efficiently implemented in hardware forexample FPGA
42 Examples of Phased-BasedMotionDetection Weappliedour motion detection algorithm on video sequences of theChange Detection 2014 dataset [22] that did not exhibitcamera egomotion For these video sequences the standarddeviation of the Gaussian window functions was set to 4
pixels and the block size was chosen to be 32 times 32 pixelsThreshold and normalization parameters were kept the samewith the exception of the ldquothermal videordquo (in order to dealwith larger background noise levels see below) We alsotested the same video sequences using the Reichardt motiondetector and the Barlow-Levick motion detector For theReichardt detector the high-pass filters were chosen as firstorder filters with a time constant of 200 milliseconds and thelow-pass filters were chosen as first order filters with a timeconstant of 300 milliseconds (assuming that the frame rate is50 frames per second)Thresholdwas set to 2 For the Barlow-Levick motion detector the time constant of the first high-pass filters was set to 250 milliseconds The low-pass filterswere the same as in the Reichardt detectorThreshold was setto 2
The first video was taken from a highway surveillancecamera (ldquohighway videordquo) under good illumination con-ditions and high contrast The video had moderate noiseparticularly on the road surface The detected motion isshown in the top left panel of Figure 9 (see SupplementaryVideo S2 for full video) The phase-based motion detectionalgorithm captured both the moving cars and the tree leavesmoving (due to the wind) In the time interval between the9th and 10th second the camera was slightly moved left-and right-wards within 5 frames again possibly due to thewind Movement due to this shift was captured by themotiondetection algorithm and the algorithm was fast enough tocorrectly determine the direction of this movement In thisvideo we already noted that this algorithm suffers from theaperture problem For example in front of the van where along horizontal edge is present the detectedmotion ismostlypointing downwards In addition to moving downwards theedge is alsomoving to the left howeverThis is expected since
Computational Intelligence and Neuroscience 13
(a) (b) (c)
Figure 9 Top motion detection of the ldquohighway videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) Red arrows indicate detected motion and its direction Bottom the contrast of the video was artificiallyreduced by 5-fold and the mean was reduced to 35 of the original The red arrows are duplicated from the motion detection result on theoriginal video as in top Blue arrows are the result of motion detection on the video with reduced contrast If motion is detected both in theoriginal video and in the video with reduced contrast then the arrow is shown in magenta (as a mix of blue and red) (see SupplementaryVideo S2 for full video)
the algorithm only detects motion locally and does not takeinto account the overall shape of any object
For comparison the motion detection results for theReichardt motion detector and the Barlow-Levick motiondetector are shown in the top middle and top right panelof Figure 9 (see Supplementary Video S2 for full video) TheReichardt detector performed relatively well when vehiclesare moving faster but the direction it predicts for vehiclesmoving slower for example on the back of the image is notaccurate In addition motion is still detected in some partsof the screen where the vehicles have just passed by Thedetection result for the Barlow-Levick motion detector waspoorer In particular the response to OFF edge movement isalways the opposite to the actual movement direction
We then squeezed the range of the screen intensity from[0 1] to [02 04] resulting in a video with a lower meanluminance and lower contrast The motion detection resultson the low-contrast video are shown in the bottom 3 panelsof Figure 9 (see Supplementary Video S2 for full video) Forreference the motion detection results for the original videoare shown in red arrows Motion detected in the low-contrastvideo is shown in blue arrows if no motion is detected forthe block in the original video If motion is detected in boththe original and the low-contrast video the arrows are shownin magenta Figure 9 clearly shows that while the motiondetection performance is degraded for all three detectors thephase-basedmotion detector performed still quite well in thelow-contrast video detecting most of the moving vehicles
The other two detectors missed many of the blocks wheremotion was detected in the original movie
To quantify how well each detector works under differentcontrast conditions we computed the ratio of unthresholdedoutput values of eachmotion detector between lower contrastand full contrast video For example in the case of phase-basedmotion detector we computed for each block the ratiobetween the PMI index for the lower contrast video and thatfor the full contrast videoThe ratios are then averaged acrossall blocks where motion is detected in the full contrast videoThis average for different contrasts is shown in Figure 10 bythe blue curve In the ideal case when the normalizationconstant 120598 is 0 the phase detector should produce invariantresponses to videos with different contrast The curve shownhere is mainly due to a nonzero 120598The ratios for the Reichardtmotion detector and the Barlow-Levick motion detectorare computed similarly and are shown in red and yellowrespectively It is clear that the phase-based motion detectorhas a superior performance across a range of contrast valuesAt 20 contrast the phase-based detector still has 50 ofthe PMI index value for full contrast video As expectedthe response of Barlow-Levick motion detector is linear withrespect to contrast and the Reichardt motion detector has aquadratic relation to contrast For a fixed threshold value alarger ratio equates to amore consistent performance at lowercontrast
The second video was captured by a surveillance camerain a train station (ldquotrain station videordquo) The video was
14 Computational Intelligence and Neuroscience
Phase-basedReichardtBarlow-Levick
0
02
04
06
08
1
Ratio
40 60 80200 100Contrast ()
Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined
under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm
The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)
The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression
With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)
43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient
objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area
For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold
We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually
We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm
We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)
These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm
5 Discussion
Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes
Computational Intelligence and Neuroscience 15
(a) (b) (c)
Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)
(a) (b) (c)
Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)
by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to
models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]
We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in
16 Computational Intelligence and Neuroscience
(a) (b) (c)
Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)
Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)
phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in
Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)
modeling in vivo biological vision systems This is consistentwith other recent findings [15]
Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure
Computational Intelligence and Neuroscience 17
(a) (b)
Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)
(a) (b)
Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)
(a) (b)
Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)
Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase
itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third
18 Computational Intelligence and Neuroscience
(a) (b)
Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)
instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection
The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions
Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors
We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well
We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It
can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes
Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems
Appendix
The Derivative of the Local Phase
Let
119886 (120596119909 120596119910 119905)
= int
R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596
119909119909 + 120596119910119910) 119889119909 119889119910
119887 (120596119909 120596119910 119905)
= minusint
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
(A1)
and therefore
11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)
= 119886 (120596119909 120596119910 119905) + 119895119887 (120596
119909 120596119910 119905)
(A2)
The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596
119909
120596119910 119905) 119886(120596
119909 120596119910 119905)) and therefore
11988912060100
119889119905
=
119886 (119889119887119889119905) minus 119887 (119889119886119889119905)
[119886]2+ [119887]2
(A3)
Computational Intelligence and Neuroscience 19
Now
119889119886
119889119905
(120596119909 120596119910 119905)
= int
R2
120597119906
120597119905
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
(A4)
For a purely translational signal with instantaneous spatialshift given by (119904
119909(119905) 119904119910(119905)) within the support of the window
we have
119889119886
119889119905
(120596119909 120596119910 119905)
= minusint
R2
119889119904119909
119889119905
(119905)
120597119906
120597119909
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus int
R2
119889119904119910
119889119905
(119905)
120597119906
120597119910
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119909
119889119904119909
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119910
119889119904119910
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199033=1199033(120596119909120596119910119905)
+ 120596119909
119889119904119909
119889119905
119887
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199034=1199034(120596119909120596119910119905)
+ 120596119910
119889119904119910
119889119905
119887
(A5)
Similarly
119889119887
119889119905
(120596119909 120596119910 119905) = minus
119889119904119909
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199035=1199035(120596119909120596119910119905)
minus 120596119909
119889119904119909
119889119905
119886 minus
119889119904119910
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199036=1199036(120596119909120596119910119905)
minus 120596119910
119889119904119910
119889119905
119886
(A6)
Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to
11988912060100
119889119905
(120596119909 120596119910 119905) =
minus (119889119904119909119889119905) [119903
3119887 + 1199035119886 + 120596119909 [
119886]2+ 120596119909 [
119887]2] minus (119889119904
119910119889119905) [119903
4119887 + 1199036119886 + 120596119910 [
119886]2+ 120596119910 [
119887]2]
[119886]2+ [119887]2
= minus
119889119904119909
119889119905
120596119909minus
119889119904119910
119889119905
120596119910minus
119889119904119909
119889119905
sdot
1199033119887 + 1199035119886
[119886]2+ [119887]2minus
119889119904119910
119889119905
sdot
1199034119887 + 1199036119886
[119886]2+ [119887]2
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
v00(120596119909120596119910119905)
(A7)
Disclosure
The authorrsquos names are alphabetically listed
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgment
The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232
References
[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010
[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010
[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014
[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010
20 Computational Intelligence and Neuroscience
[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013
[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014
[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956
[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010
[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985
[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965
[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981
[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015
[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014
[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999
[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015
[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001
[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996
[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980
[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013
[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013
[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014
[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014
[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach
to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012
[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990
[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987
[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998
Submit your manuscripts athttpwwwhindawicom
Computer Games Technology
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Distributed Sensor Networks
International Journal of
Advances in
FuzzySystems
Hindawi Publishing Corporationhttpwwwhindawicom
Volume 2014
International Journal of
ReconfigurableComputing
Hindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Applied Computational Intelligence and Soft Computing
thinspAdvancesthinspinthinsp
Artificial Intelligence
HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014
Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Journal of
Computer Networks and Communications
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation
httpwwwhindawicom Volume 2014
Advances in
Multimedia
International Journal of
Biomedical Imaging
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
ArtificialNeural Systems
Advances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience
Industrial EngineeringJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Human-ComputerInteraction
Advances in
Computer EngineeringAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
2 Computational Intelligence and Neuroscience
Algorithm
In vivo neural circuit
In silicoDSP chip
Figure 1 Algorithms can have different physical realizations Forexample an algorithm can be implemented by neural circuits in abiological system Alternately it can be implemented on a digitalsignal processor Algorithms direct the implementation on thephysical layer Conversely biological neural circuits inspire newalgorithmic designs which can in turn be expressed and improvedupon by a realization in silico
time optic flow estimation algorithms produce accurateresults the computational demand to perform many of thesealgorithms is too high for real-time implementation
Several models for biological motion detection are avail-able and their architecture is quite simple [6] The Reichardtmotion detector [7] was thought to be the underlyingmodel for motion detection in insects [8] The model isbased on a correlation method to extract motion inducedby spatiotemporal information patterns of light intensityTherefore it relies on a correlationmultiplication operationA second model is the motion energy detector [9] It usesspatiotemporal separable filters and a squaring nonlinearityto compute motion energy and it was shown to be equivalentto the Reichardt motion detector Earlier work in the rabbitretina was the foundation to the Barlow-Levick model [10]of motion detection The model relies on inhibition tocompensate motion in the null direction
In this paper we provide an alternative motion detectionalgorithm based on local phase information of the visualscene Similar to mechanisms in other biological models itoperates in continuous time and in parallel Moreover themotion detection algorithm we propose can be efficientlyimplemented on parallel hardware This is again similar tothe properties of biological motion detection systems Ratherthan focusing on velocity of motion we focus on localizationthat is where the motion occurs in the visual field as well asthe direction of motion
It has been shown that images can be represented bytheir global phase alone [11] Here we provide a reconstruc-tion algorithm of visual scenes by only using local phaseinformation thereby demonstrating the spectrum of therepresentational capability of phase information
The Fourier shift property clearly suggests the relation-ship between the global shift of an image and the global phaseshift in the frequency domain We elevate this relationshipby computing the change of local phase to indicate motionthat appears locally in the visual scene The local phases arecomputed using window functions that tile the visual fieldwith overlapping segments making it amenable for a highlyparallel implementation In addition we propose a Radon-transform-based motion detection index on the change of
local phases for the readout of the relation between localphases and motion
Interestingly phase information has been largely ignoredin the field of linear signal processing and for good reasonPhase-based processing is intrinsically non-linear Recentresearches however showed that phase information canbe smartly employed in speech processing [12] and visualprocessing [13] For example spatial phase in an image isindicative of local features such as edges when consideringphase congruency [14] The role of spatial phase in compu-tational and biological vision emergence of visual illusionsandpattern recognition is discussed in [15] Togetherwith ourresult for motion detection these studies suggest that phaseinformation has a great potential for achieving efficient visualsignal processing
This paper is organized as follows In Section 2 weshow that local phase information can be used to faithfullyrepresent visual information in the absence of amplitudeinformation In Section 3 we develop a simple local phase-based algorithm to detectmotion in visual scenes and providean efficient way for its implementation We then provideexamples and applications of the proposed motion detectionalgorithm to motion segmentation We also compare ourresults to those obtained using motion detection algorithmsbased on optic flow techniques in Section 4 Finally wesummarize our results in Section 5
2 Representation of Visual Scenes UsingPhase Information
Theuse of complex valued transforms is widespread for bothrepresenting and processing images When represented inpolar coordinates the output of a complex valued transformof a signal can be split into amplitude and phase In thissection we define two types of phases of an image globalphase and local phase We then argue that both types ofphases can faithfully represent an image and we provide areconstruction algorithm that recovers the image from localphase information This indicates that phase informationalone can largely represent an image or video signal
It will become clear in the sections that follow that theuse of phase for representing of image and video infor-mation leads to efficient ways to implement certain typesof imagevideo processing algorithms for example motiondetection algorithms
21 The Global Phase of Images Recall that the Fouriertransform of a real valued image 119906 = 119906(119909 119910) (119909 119910) isin R2is given by
(120596119909 120596119910) = int
R2119906 (119909 119910) 119890
minus119895(120596119909119909+120596119910119910)119889119909 119889119910 (1)
with (120596119909 120596119910) isin C and (120596
119909 120596119910) isin R2
In polar coordinates the Fourier transform of 119906 can beexpressed as
(120596119909 120596119910) = (120596
119909 120596119910) 119890119895(120596119909120596119910) (2)
Computational Intelligence and Neuroscience 3
where (120596119909 120596119910) isin R is the amplitude and
120601(120596119909 120596119910) is the
phase of the Fourier transform of 119906
Definition 1 The amplitude of the Fourier transform of animage 119906 = 119906(119909 119910) (119909 119910) isin R2 is called the global amplitudeof 119906 The phase of the Fourier transform of an image 119906 =
119906(119909 119910) (119909 119910) isin R2 is called the global phase of 119906
It is known that the global phase of an image plays animportant role in the representation of natural images [11] Aclassic example is to take two images and to exchange theirglobal phases before their reconstruction using the inverseFourier transform The resulting images are slightly smearedbut largely reflect the information contained in the globalphase
22The Local Phase of Images The global phase indicates theoffset of sinusoids of different frequencies contained in theentire image However it is not intuitive to relate the globalphase to local image features such as edges and their positionin an image To study these local features it is necessary tomodify the Fourier transform such that it reflects propertiesof a restricted region of an image It is natural to consider theShort-Time (-Space) Fourier Transform (STFT)
119880(120596119909 120596119910 1199090 1199100) = int
R2119906 (119909 119910)119908 (119909 minus 119909
0 119910 minus 119910
0)
sdot 119890minus119895(120596119909(119909minus1199090)+120596119910(119910minus1199100))119889119909 119889119910
(3)
where 119908 = 119908(119909 119910) (119909 119910) isin R2 is a real valued windowfunction centered at (119909
0 1199100) isin R2 Typical choices of window
functions include the Hann window and the Gaussian win-dow The (effectively) finite support of the window restrictsthe Fourier transform to local image analysis
Similarly to the Fourier transform the STFT can beexpressed in polar coordinates as
119880(120596119909 120596119910 1199090 1199100) = 119860 (120596
119909 120596119910 1199090 1199100) 119890119895120601(12059611990912059611991011990901199100) (4)
where 119860(120596119909 120596119910 1199090 1199100) isin R is the amplitude and 120601(120596
119909 120596119910
1199090 1199100) is the phase of the STFT
Definition 2 The amplitude of the STFT of an image 119906 =
119906(119909 119910) (119909 119910) isin R2 is called the local amplitude of 119906 Thephase of the STFT of an image 119906 = 119906(119909 119910) (119909 119910) isin R2 iscalled the local phase of 119906
Note that when 119908 is a Gaussian window the STFT of 119906evaluated at (120596
119909 120596119910 1199090 1199100) can be equivalently viewed as the
response of a complex-valued Gabor receptive field
ℎ (119909 119910) = 119890minus((119909minus119909
0)2+(119910minus119910
0)2)21205902
119890minus119895(120596119909(119909minus1199090)+120596119910(119910minus1199100)) (5)
to 119906In this case the window of the STFT clearly is given by
119908 (119909 119910) = 119890minus(1199092+1199102)21205902
(6)
Therefore the STFT can be realized by an ensemble of Gaborreceptive fields that are common in modeling simple andcomplex cells (neurons) in the primary visual cortex [16]
23 Reconstruction of Images from Local Phase Amplitudeand phase can be interpreted as measurementsprojectionsof images that are indicative of their information contentClassically when both the global amplitude and phase areknown it is straightforward to reconstruct the image Thereconstruction calls for computing the inverse Fourier trans-form given the global amplitude and phase Similarly whenusing local amplitude and phase if the sampling functionsform a basis or frame in a space of images the reconstructionis provided by the formalism of wavelet theory for exampleusing Gabor wavelets [17]
Amplitude or phase represents partial informationextracted from visual scenesThey are obtained via nonlinearsampling that is a nonlinear operation for extracting theamplitude andphase information from imagesThenonlinearoperation makes reconstruction from either amplitude orphase alone difficult Earlier studies and recent developmentin solving quadratic constraints however suggest that it ispossible to reconstruct images from global or local amplitudeinformation [18 19]
While computing the amplitude requires a second order(quadratic) nonlinearity computing the phase calls for higherorder nonlinear operators (Volterra kernels) It is possiblehowever to reconstruct up to a constant scale an image fromits global phase information alone without explicitly usingthe amplitude information An algorithm was provided forsolving this problem in the discrete signal processing domainin [11]The algorithm smartly avoids using the inverse tangentfunction by reformulating the phase measurements as a set oflinear equations that are easy to solve
Using a similar argument we demonstrate in the follow-ing that up to a constant scale a bandlimited signal 119906 =
119906(119909 119910) (119909 119910) isin R2 can be reconstructed from its local phasealone We first formulate the encoding of an image 119906 by localphase using the Gabor receptive fields as a special case Itis straightforward then to formulate the problem with localphase computed from other types of STFTs
Formally we consider an image 119906 = 119906(119909 119910) on thedomain R2 to be an element of a space of trigonometricpolynomialsH of the form
119906 (119909 119910) =
119871119909
sum
119897119909=minus119871119909
119871119910
sum
119897119910=minus119871119910
119888119897119909119897119910
119890119897119909119897119910
(119909 119910) (7)
where
119890119897119909119897119910
= exp(119895119897119909
Ω119909
119871119909
119909) exp(119895119897119910
Ω119910
119871119910
119910)
119897119909= minus119871119909 119871
119909 119897119910= minus119871119910 119871
119910
(8)
are the set of basis functions ofHΩ119909 119871119909are the bandwidth
and the order respectively of the space in the 119909 dimensionand Ω
119910 119871119910are the bandwidth and the order respectively of
the space in the 119910 dimensionH is a Reproducing Kernel Hilbert Space with inner
product [20 21]
⟨1199061 1199062⟩ = int
[0119879119909]times[0119879
119910]
1199061(119909 119910) 119906
2(119909 119910)119889119909 119889119910 (9)
4 Computational Intelligence and Neuroscience
where 119879119909= 2120587119871
119909Ω119909 119879119910= 2120587119871
119910Ω119910are the period of the
space in the 119909 and 119910 dimensions respectivelyConsider a bank of119873 Gabor receptive fields
ℎ119896119897119898119899
(119909 119910) = T119896119897(119908 (119909 119910) 119890
minus119895(120596119909119898119909+120596119910119899119910)) (10)
where T119896119897is the translation operator with T
119896119897119906 = 119906(119909 minus
1198961198870 119910 minus 119897119887
0) 119896 119897 isin Z 119887
0gt 0 0 le 119896119887
0le 119879119909 and 0 le
1198971198870le 119879119910 and 120596
119909119898
= 1198981205960 120596119910119899
= 1198991205960 119898 119899 isin Z 120596
0gt 0
minusΩ119909le 119898120596
0le Ω119909 and minusΩ
119910le 1198991205960le Ω119910 The responses of
the Gabor receptive fields to the input 119906(119909 119910) are given by
int
R2119906 (119909 119910) ℎ
119896119897119898119899(119909 119910) 119889119909 119889119910 = int
R2119906 (119909 119910)
sdot 119908 (119909 minus 1198961198870 119910 minus 119897119887
0) 119890minus119895(120596119909119898(119909minus119896119887
0)+120596119910119899(119910minus1198971198870))119889119909 119889119910
= 119860119896119897119898119899
119890119895120601119896119897119898119899
(11)
where 119860119896119897119898119899
ge 0 119860119896119897119898119899
isin R is the local amplitude and120601119896119897119898119899
isin [0 2120587) is the local phaseDividing both sides of (11) by 119890119895120601119896119897119898119899 we have
int
R2119906 (119909 119910)119908 (119909 minus 119896119887
0 119910 minus 119897119887
0)
sdot 119890minus119895(120596119909119898(119909minus119896119887
0)+120596119910119899(119910minus1198971198870)+120601119896119897119898119899)119889119909 119889119910 = 119860
119896119897119898119899
(12)
Since 119860119896119897119898119899
isin R we have
int
R2119906 (119909 119910)119908 (119909 minus 119896119887
0 119910 minus 119897119887
0)
sdot cos (120596119909119898
(119909 minus 1198961198870) + 120596119910119899
(119910 minus 1198971198870) + 120601119896119897119898119899
) 119889119909 119889119910
= 119860119896119897119898119899
(13)
int
R2119906 (119909 119910)119908 (119909 minus 119896119887
0 119910 minus 119897119887
0)
sdot sin (120596119909119898
(119909 minus 1198961198870) + 120596119910119899
(119910 minus 1198971198870) + 120601119896119897119898119899
) 119889119909 119889119910
= 0
(14)
Note that119908(119909 minus 1198961198870 119910 minus 119897119887
0) sin(120596
119909119898
(119909 minus 1198961198870) + 120596119910119899
(119910 minus 1198971198870) +
120601119896119897119898119899
) is a real-valued Gabor receptive field with a preferredphase at 120601
119896119897119898119899+ 1205872 minus 120596
119909119898
1198961198870minus 120596119910119899
1198971198870
Remark 3 Assuming that the local phase information 120601119896119897119898119899
is obtained via measurements that is filtering the image 119906
with pairs of Gabor receptive fields (10) the set of linearequations (14) has a simple interpretation the image 119906 isorthogonal to the space spanned by the functions
119908 (119909 minus 1198961198870 119910 minus 119897119887
0)
sdot sin (120596119909119898
(119909 minus 1198961198870) + 120596119910119899
(119910 minus 1198971198870) + 120601119896119897119898119899
)
(15)
where (119896 119897 119898 119899) isin I with I = (119896 119897 119898 119899) isin Z4 | 0 le 1198961198870le
119879119909 0 le 119897119887
0le 119879119910 minusΩ119909le 119898120596
0le Ω119909 minusΩ119910le 1198991205960le Ω119910
We are now in the position to provide a reconstructionalgorithm of the image from phase 120601
119896119897119898119899 (119896 119897 119898 119899) isin I
Lemma 4 119906 can be reconstructed from 120601119896119897119898119899
(119896 119897 119898 119899) isin Ias
119906 (119909 119910) =
119871119909
sum
119897119909=minus119871119909
119871119910
sum
119897119910=minus119871119910
119888119897119909119897119910
119890119897119909119897119910
(119909 119910) (16)
with
Φ119888 = 0 (17)
whereΦ is a matrix whose 119901th row and 119902th column entry are
[Φ]119901119902
= int
R2119908 (119909 minus 119896119887
0 119910 minus 119897119887
0)
sdot sin (120596119909119898
(119909 minus 1198961198870) + 120596119910119899
(119910 minus 1198971198870) + 120601119896119897119898119899
)
sdot 119890119897119909119897119910
(119909 119910) 119889119909 119889119910
(18)
Here 119901 traverses the set I and 119902 = (2119871119910+ 1)(119897119909+ 119871119909) + (119897119910+
119871119910+ 1) 119888 is a vector of the form
119888 = [119888minus119871119909minus119871119910
119888minus119871119909minus119871119910+1 119888
minus119871119909119871119910
119888minus119871119909+1minus119871
119910
119888minus119871119909+1minus119871
119910+1 119888
minus119871119909+1119871119910
119888119871119909minus119871119910
119888119871119909minus119871119910+1
119888119871119909119871119910
]
119879
(19)
that belongs to the null space of Φ A necessary conditionfor perfect reconstruction of 119906 up to a constant scale is that119873 ge (2119871
119909+ 1)(2119871
119910+ 1) minus 1 where 119873 is the number of phase
measurements
Proof Substituting (7) into (14) we obtain119871119909
sum
119897119909=minus119871119909
119871119910
sum
119897119910=minus119871119910
119888119897119909119897119910
int
R2119908 (119909 minus 119896119887
0 119910 minus 119897119887
0)
sdot sin (120596119909119898
(119909 minus 1198961198870) + 120596119910119899
(119910 minus 1198971198870) + 120601119896119897119898119899
)
sdot 119890119897119909119897119910
(119909 119910) 119889119909 119889119910 = 0
(20)
for all 119896 119897 119898 119899 isin Z Therefore we have
Φ119888 = 0 (21)
inferring that 119888 is in the null space ofΦIf 119873 lt (2119871
119909+ 1)(2119871
119910+ 1) minus 1 it follows from the rank-
nullity theorem that dim(null(Φ)) gt 1 leading to multiplelinearly independent solutions toΦ119888 = 0
Example 5 In Figure 2 an example of reconstruction of animage is shown using only local phase information Thereconstructed signal was scaled to match the original signalThe SNR of the reconstruction is 4448 [dB] An alternativeway to obtain a unique reconstruction is to include anadditional measurement for example the mean value of thesignal int
R2119906(119909 119910)119889119909 119889119910 to the system of linear equations (14)
Computational Intelligence and Neuroscience 5
(a) (b) (c)
Figure 2 An example of reconstruction of image from local phase information (a) Original image the dimension of the space is 16129(b) Reconstructed image from 20102 phase measurements scaled to match the original (c) Error
3 Visual Motion Detection fromPhase Information
In this section we consider visual fields that change as afunction of time For notational simplicity 119906 will denote herethe space-time intensity of the visual field
31 The Global Phase Equation for Translational Motion Let119906 = 119906(119909 119910 119905) (119909 119910) isin R2 119905 isin R be a visual stimulus If thevisual stimulus is a pure translation of the signal at 119906(119909 119910 0)that is
119906 (119909 119910 119905) = 119906 (119909 minus 119904119909(119905) 119910 minus 119904
119910(119905) 0) (22)
where
119904119909(119905) = int
119905
0
V119909(119904) 119889119904
119904119910(119905) = int
119905
0
V119910(119904) 119889119904
(23)
are the total length of translation at time 119905 in each dimen-sion and V
119909(119905) and V
119910(119905) are the corresponding instanta-
neous velocity components then the only difference between119906(119909 119910 119905) and 119906(119909 119910 0) in the Fourier domain is captured bytheir global phase More formally consider the following
Lemma 6 The change (derivative) of the global phase is givenby
119889120601 (120596119909 120596119910 119905)
119889119905
= minus120596119909V119909(119905) minus 120596
119910V119910(119905)
= minus [120596119909 120596119910] [V119909(119905) V119910(119905)]
119879
(24)
where by abuse of notation 120601(120596119909 120596119910 119905) denotes the global
phase of 119906(119909 119910 119905) and 120601(120596119909 120596119910 0) is the initial condition
Proof If (F119906(sdot sdot 0)) (120596119909 120596119910) is the 2D (spatial) Fourier
transform of 119906 = 119906(119909 119910 0) (119909 119910) isin R2 by the Fourier shifttheorem we have
(F119906 (sdot sdot 119905)) (120596119909 120596119910)
= (F119906 (sdot sdot 0)) (120596119909 120596119910) 119890minus119895(120596119909119904119909(119905)+120596
119910119904119910(119905))
(25)
For a certain frequency component (120596119909 120596119910) the change of its
global phase over time amounts to
119889120601 (120596119909 120596119910 119905)
119889119905
= minus120596119909
119889119904119909(119905)
119889119905
minus 120596119910
119889119904119910(119905)
119889119905
= minus120596119909V119909(119905) minus 120596
119910V119910(119905)
= minus [120596119909 120596119910] [V119909(119905) V119910(119905)]
119879
(26)
Therefore in the simple case where the entire visual field isshifting the derivative of the phase of Fourier componentsindicates motion and it can be obtained by the innerproduct between the component frequency and the velocityvector
32 The Change of Local Phase
321The Local Phase Equation for Translational Motion Theanalysis in Section 31 applies to global motion This type ofmotion occurs most frequently when the imaging deviceeither an eye or a cameramoves Visualmotion in the naturalenvironment however ismore diverse across the screen sinceit is often time produced by multiple moving objects Theobjects can be small and the motion more localized in thevisual field
Taking the global phase of 119906 will not simply revealwhere motion of independent objects takes place or theirdirectionvelocity of motion The ease of interpretation ofmotion by using the Fourier transform however motivates
6 Computational Intelligence and Neuroscience
us to reuse the same concept in detecting local motion Thiscan be achieved by restricting the domain of the visual fieldwhere the Fourier transform is applied
To be able to detect local motion we consider thelocal phase of 119906(119909 119910 119905) by taking the STFT with windowfunction 119908(119909 119910) Note that the STFT and its ubiquitousimplementation in DSP chips can be extensively used in anydimension For simplicity and without loss of generality weconsider the window to be centered at (0 0) The STFT isgiven by
int
R2119906 (119909 119910 119905) 119908 (119909 119910) 119890
minus119895(120596119909119909+120596119910119910)119889119909 119889119910
= 11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)
(27)
where by abuse of notation 11986000(120596119909 120596119910 119905) is the amplitude
and 12060100(120596119909 120596119910 119905) the local phase
Before we move on to the mathematical analysis we canintuitively explain the relation between the change in localphase and visual motion taking place across the windowsupport First if the stimulus undergoes a uniform changeof intensity or it changes proportionally over time due tolighting conditions for example the local phase does notchange since the phase is invariant with respect to intensityscaling Therefore the local phase does not change for suchnonmotion stimuli Second a rigid edge moving across thewindow support will induce a phase change
For a strictly translational signal within the windowsupport (footprint) for example
119906 (119909 119910 119905) = 119906 (119909 minus 119904119909(119905) 119910 minus 119904
119910(119905) 0)
for (119909 119910) isin supp (119908 (119909 119910))
(28)
where 119904119909(119905) and 119904
119910(119905) are as defined in (23) we have the
following result
Lemma 7 Consider the following
11988912060100
119889119905
(120596119909 120596119910 119905) = minus
119889119904119909(119905)
119889119905
120596119909minus
119889119904119910(119905)
119889119905
120596119910
+ v00(120596119909 120596119910 119905)
(29)
where by abuse of notation 12060100(120596119909 120596119910 119905) is the local phase
of 119906(119909 119910 119905) and 12060100(120596119909 120596119910 0) is the initial condition The
functional form of the term v00
= v00(120596119909 120596119910 119905) is provided
in the Appendix
Proof The derivation of (29) and the functional form of v00
are given in the Appendix
Remark 8 Wenotice that the derivative of the local phase hassimilar structure to that of the global phase but for the addedterm v
00Through simulations we observed that the first two
terms in (29) dominate over the last term for an ON or OFFmoving edge [3] For example Figure 3 shows the derivativeof the local phase given in (29) for an ON edge moving withvelocity (40 0) pixelssec
42
0minus2
minus4
100
d120601dt
120596yminus250minus200minus150minus100minus50
050
150
200250
31 2minus4 0minus1minus3 minus2 4
120596x
Figure 3 Derivative of the local phase for an ON edge moving inthe positive 119909 direction
Remark 9 Note that 12060100(120596119909 120596119910 119905) may not be differen-
tiable even if 119906(119909 119910 119905) is differentiable particularly when11986000(120596119909 120596119910 119905) = 0 For example the spatial phase can jump
from a positive value to zero when11986000(120596119909 120596119910 119905) diminishes
This also suggests that the instantaneous local spatial phaseis less informative about a region of a visual scene whenever11986000(120596119909 120596119910 119905) is close to zero Nevertheless the time deriva-
tive of the local phase can be approximated by applying ahigh-pass filter to 120601
00(120596119909 120596119910 119905)
322 The Block Structure for Computing the Local Phase Weconstruct Gaussian windows along the 119909 119910 dimensions TheGaussian windows are defined as
(T119896119897119908) (119909 119910) = 119890
minus((119909minus119909119896)2+(119909minus119910
119897)2)21205902
(30)
where 119909119896= 1198961198870 119910119897= 1198971198870 in which 119887
0isin Z+ is the distance
between two neighboring windows and 1 le 1198961198870le 119875119909 1 le
1198971198870le 119875119910 where 119875
119909 119875119910isin Z+ are the number of pixels of the
screen in 119909 and 119910 directions respectivelyWe then take the 2D Fourier transform of the windowed
video signal 119906(119909 119910 119905)(T119896119897119908)(119909 119910) and write in polar form
int
R2119906 (119909 119910 119905) (T
119896119897119908) (119909 119910) 119890
minus119895(120596119909(119909minus119909119896)+120596119910(119910minus119910119897))119889119909 119889119910
= 119860119896119897(120596119909 120596119910 119905) 119890119895120601119896119897(120596119909120596119910119905)
(31)
The above integral can be very efficiently evaluated usingthe 2D FFT in discrete domain defined on 119872 times 119872 blocksapproximating the footprint of the Gaussian windows Forexample the standard deviation of the Gaussian windows weuse in the examples in Section 4 is 4 pixels A block of 32times32
pixels (119872 = 32) is sufficient to cover the effective support(or footprint) of the Gaussian window At the same time thesize of the block is a power of 2 which is most suitable forFFT-based computationThe processing of each block (119896 119897) isindependent of all other blocks thereby parallelism is readilyachieved
Note that the size of the window is informed by thesize of the objects one is interested in locating Measure-ments of the local phase using smaller window functions
Computational Intelligence and Neuroscience 7
60
50
40
30
20
10
y (pi
xel)
01
02
03
04
05
06
07
08
09
1
20 30 40 50 6010x (pixel)
(a)
0
02
04
06
08
1
Mag
nitu
de (a
u)
10 20 30 40 50 60 700y (pixel)
(b)
Figure 4 Example of block structure (a) Four yellow disks show 4 Gaussian window functions with translation parameter 119896 = 3 119897 = 3
(top-left) 119896 = 3 119897 = 7 (bottom-left) 119896 = 7 119897 = 3 (top-right) and 119896 = 7 119897 = 7 (bottom-right) The red solid square shows a 32 times 32-pixel blockapproximating the Gaussian window with 119896 = 3 119897 = 3 and the green dashed square shows a 32 times 32-pixel block approximating the Gaussianwindow with 119896 = 3 119897 = 7 (b) Cross section of all 11 Gaussian windows with translation parameters 119896 = 7 119897 isin [0 10] The cross section istaken as indicated by the magenta line in (a) and the red and blue curve correspond to cross sections of the two Gaussian windows shown in(a) centered on the magenta line
are less robust to noise Larger windows would enhanceobject motion detection if the object size is comparable tothe window size However there would be an increasedlikelihood of independent movement of multiple objectswithin the same window which is not modeled here andthereby may not be robustly detected
Therefore for each block (119896 119897) we obtain 1198722 measure-
ments of the phase 120601119896119897(120596119909119898
120596119910119898
119905) at every time instant 119905with (120596
119909119898
120596119910119898
) isin D2 where
D2= (120596
119909119898
= 1198981205960 120596119910119899
= 1198991205960) 119898 119899 = minus1198722
minus 1198722 + 1 1198722 minus 1
(32)
with 1205960= 2120587119872 We then compute the temporal derivative
of the phase that is (119889120601119896119897119889119905)(120596
119909 120596119910 119905) for (120596
119909 120596119910) isin D2
We further illustrate an example of the block structurein Figure 4 Figure 4(a) shows an example of an image of64 times 64 pixels Four Gaussian windows are shown each witha standard deviation of 4 pixels The distance between thecenters of two neighboring Gaussian windows is 6 pixelsThe red solid square shows a 32 times 32-pixel block with 119896 =
3 119897 = 3 which encloses effective support of the Gaussianwindow on top-left (119896 = 0 119897 = 0 is the block with Gaussianwindow centered at pixel (1 1)) The green dashed squareshows another 32 times 32-pixel block with 119896 = 3 119897 = 7 Thetwo Gaussian windows on the right are associated with theblocks 119896 = 7 119897 = 3 and 119896 = 7 119897 = 7 respectively Crosssection of all Gaussian windows with 119896 = 7 119897 isin [0 10] that isthose centered on themagenta line are shown in Figure 4(b)The red and blue curve in Figure 4(b) correspond to the two
Gaussian windows shown in Figure 4(a) Figure 4(b) alsosuggests that some of the Gaussian windows are cut off on theboundaries This is however equivalent to assuming that thepixel values outside the boundary are always zero and it willnot significantly affect motion detection based on the changeof local phase
Since the phase and thereby the phase change is noisierwhen the local amplitude is low an additional denoisingstep can be employed to discount the measurements of(119889120601119896119897119889119905)(120596
119909 120596119910 119905) for low amplitude values 119860
119896119897(120596119909 120596119910 119905)
The denoising is given by
119889120601119896119897
119889119905
(120596119909 120596119910 119905)
sdot
119860119896119897(120596119909 120596119910 119905)
(11198722)sum(120596119909120596119910)isinD2 119860119896119897 (120596119909 120596119910 119905) + 120598
(33)
where 120598 gt 0 is a constant and (120596119909 120596119910) isin D2
33 The Phase-Based Detector We propose here a block FFTbased algorithm to detect motion using phase informationSuch an algorithm is due to its simplicity and parallelismhighly suitable for an in silico implementation
331 Radon Transform on the Change of Phases We exploitthe approximately linear structure of the phase derivative forblocks exhibiting motion by computing the Radon transformof (119889120601
119896119897119889119905)(120596
119909 120596119910 119905) over a circular bounded domain 119862 =
(120596119909 120596119910) | (120596119909 120596119910) isin D2 1205962
119909+ 1205962
119910lt 1205872
8 Computational Intelligence and Neuroscience
The Radon transform of the change of phase in thedomain 119862 is given by
(R119889120601119896119897
119889119905
) (120588 120579 119905) = int
R
119889120601119896119897
119889119905
(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588
sdot sin 120579 + 119904 sdot cos 120579 119905) 1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579
+ 119904 sdot cos 120579) 119889119904
(34)
where
1119862(120596119909 120596119910) =
1 if (120596119909 120596119910) isin 119862
0 otherwise(35)
The Radon transform (R(119889120601119896119897119889119905))(120588 120579 119905) evaluated at
a particular point (1205880 1205790 1199050) is essentially an integral
of (119889120601119896119897119889119905)(120596
119909 120596119910 1199050) along a line oriented at angle
1205872 + 1205790with the 120596
119909axis and at distance |120588
0| along the
(cos(1205790) sin(120579
0)) direction from (0 0)
If for a particular 119896 and 119897 we have (119889120601119896119897119889119905)(120596
119909 120596119910 119905) =
minusV119909(119905)120596119909minus V119910(119905)120596119910 we have
(R (119889120601119896119897119889119905)) (120588 120579 119905)
c (120588 120579)
= 120588 [minusV119909(119905) cos 120579 minus V
119910(119905) sin 120579]
(36)
where
c (120588 120579)
= int
R
1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579 + 119904 sdot cos 120579) 119889119904
(37)
c(120588 120579) is a correction term due to different length of lineintegrals for different values of (120588 120579) in the bounded domain119862
After computing the Radon transform of (119889120601119896119897119889119905)(120596
119909
120596119910 119905) for every block (119896 119897) at time 119905
0 we compute the Phase
Motion Indicator (PMI) defined as
PMI119896119897= max120579isin[0120587)
sum
120588
100381610038161003816100381610038161003816100381610038161003816
(R (119889120601119896119897119889119905)) (120588 120579 119905
0)
c (120588 120579)
100381610038161003816100381610038161003816100381610038161003816
(38)
If the PMI119896119897
is larger than a chosen threshold motion isdeemed to occur in block (119896 119897) at time 119905
0
Using the Radon transform makes it easier to separaterigid motion from noise Since the phase is quite sensitiveto noise particularly when the amplitude is very smallthe change of phase under noise may have comparablemagnitude to that due to motion as mentioned earlier Thechange of phase under noise however does not possess thestructure suggested by (29) in the (120596
119909 120596119910) domain Instead
it appears to be more randomly distributed Consequently
the PMI value is comparatively small for these blocks (see alsoSection 333)
Moreover the direction of motion for block (119896 119897) wheremotion is detected can be easily computed as
120579119896119897
= 120587(
sign (sum120588gt0
(R (119889120601119896119897119889119905)) (120588 120572
119896119897 1199050) c (120588 120572
119896119897)) + 1
2
)
+ 120572119896119897
(39)
where
120572119896119897= argmax120579isin[0120587)
sum
120588
100381610038161003816100381610038161003816100381610038161003816
(R (119889120601119896119897119889119905)) (120588 120579 119905
0)
c (120588 120579)
100381610038161003816100381610038161003816100381610038161003816
(40)
This follows from (36)
332 The Phase-Based Motion Detection Algorithm We for-mally summarize the above analysis as Algorithm 1 Figure 5shows a schematic diagram of the proposed phase-basedmotion detection algorithm
The algorithm is subdivided into two parts The first partcomputes local phase changes and the second part is thephase-based motion detector
In the first part the screen is divided into overlappingblocks For example the red green and blue blocks in theplane ldquodivide into overlapping blocksrdquo correspond to thesquares of the same color covering the video stream AGaussian window is then applied on each block followed bya 2D FFT operation that is used to extract the local phase Atemporal high-pass filter is then employed to extract phasechanges
In the second part the PMI is evaluated for each blockbased on the Radon transform of the local phase changes ineach block Motion is detected for blocks with PMI largerthan a preset threshold and the direction of motion iscomputed as in (39)
It is easy to notice that the algorithm can be highlyparallelized
333 Example We provide an illustrative example inFigure 6 showing how motion is detected using Algorithm 1The full video of this example can be found in SupplementaryVideo S1 see Supplementary Material available online athttpdxdoiorg10115520167915245 Figure 6(a) depicts astill from the ldquohighway videordquo in the Change Detection 2014dataset [22] evaluated at a particular time 119905
0 As suggested
by the algorithm the screen in Figure 6(a) is divided into26 times 19 overlapping blocks and the window functions areapplied to each block Local phases can then be extractedfrom the 2DFFT of eachwindowed block and the local phasechanges are obtained by temporal high-pass filtering Thephase change is shown in Figure 6(b) for all blocks with block(12 11) enlarged in Figure 6(c) and block (23 6) enlarged inFigure 6(d) (see also the plane ldquo2D FFT and extract phasechangerdquo in Figure 5) Note that at the time of the videoframe block (12 11) covers a part of the vehicle in motion
Computational Intelligence and Neuroscience 9
Input Visual stream or Video data 119906(119909 119910 119905)Output Detected MotionConstruct Gaussian window of size119872times119872 denote as 119908for 119905 = Δ119879 2Δ119879 119873Δ119879 do
Divide the screen of frame at time 119905 into multiple119872times119872 blocks 119906119896119897(119905) with overlaps
foreach block 119906119896119897(119905) do
Multiply pixel-wise the block 119906119896119897(119905) with the Gaussian window 119906
119896119897(119905)119908
Take119872times119872 2D FFT of 119906119896119897(119905)119908 denote phase as 120601
119896119897(120596119909 120596119910 119905) and amplitude as 119860
119896119897(120596119909 120596119910 119905)
foreach frequency (120596119909 120596119910) do
Highpass filter temporally 120601119896119897(120596119909 120596119910 119905) denote 1206011015840
119896119897(120596119909 120596119910 119905)
Optionally denoise by 1206011015840119896119897(120596119909 120596119910 119905) larr 120601
1015840
119896119897(120596119909 120596119910 119905)(119860
119896119897(120596119909 120596119910 119905)((1119872
2) sum(120596119909 120596119910)isinD
2 119860119896119897(120596119909 120596119910 119905) + 120598))
endCompute the radon transform (R120601
1015840
119896119897)(120588 120579 119905)
Compute PMI119896119897(119905) according to (38)
if PMI119896119897(119905) gt threshold then
Motion Detected at block (119896 119897) at time 119905Compute direction of motion according to (39)
endend
end
Algorithm 1 Phase-based motion detection algorithm using the FFT
y
120596x
120596y
y
t
120596x
120596y
x
y
x
y
x
x
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
Divided intooverlapping blocks
Apply window function
2D FFT andextract phase change
Radon transformand extract strength
and orientation of plane
Thresholding
Computephase
change
The phase-baseddetector
Figure 5 Schematic diagram of proposed phase-based motion detection algorithm
10 Computational Intelligence and Neuroscience
in the front and block (23 6) corresponds to an area of thehighway pavement where no motion occurs
Figure 6(f) depicts for each block (119896 119897) the maximumphase change over all (120596
119909 120596119910) isin D2 that is
max(120596119909120596119910)isinD2
10038161003816100381610038161003816100381610038161003816
119889120601119896119897
119889119905
(120596119909 120596119910 1199050)
10038161003816100381610038161003816100381610038161003816
(41)
We observe from the figure that for regions with lowamplitude such as the region depicting the road whenthe normalization constant is absent the derivative of thephase can be noisy For these blocks the maximum of|(119889120601119896119897119889119905)(120596
119909 120596119910 1199050)| over all (120596
119909 120596119910) isin D2 is comparable
to the maximum obtained for blocks that cover the vehiclesin motion
However (29) suggests that the local phase change frommultiple filter pairs centered at the same spatial position(119896 119897) can provide a constraint to robustly estimate motionand its direction Given the block structure employed in thecomputation of the local phase it is natural to utilize phasechange information from multiple sources
Indeed if for a particular block (119896 119897) (119889120601119896119897119889119905)(120596
119909
120596119910 119905) = minusV
119909(119905)120596119909minus V119910(119905)120596119910 then it is easy to see that (119889120601
119896119897
119889119905)(120596119909 120596119910 119905) will be zero on the line V
119909(119905)120596119909+ V119910(119905)120596119910= 0
and have opposite sign on either side of this line For examplein Figures 6(b) and 6(c) (119889120601
119896119897119889119905)(120596
119909 120596119910 1199050) clearly exhibits
this property for blocks that cover a vehicle in motion ThePMI is a tool to evaluate this property
Finally the PMIs for all blocks are shown compactlyin a heat map in Figure 6(e) The figure shows clearly thatthe blocks corresponding to the two moving vehicles have ahigh PMI value while the stationary background areas havea low PMI value allowing one to easily detect motion byemploying simple thresholding (see also the plane ldquoRadontransform and extract strength orientation of planerdquo inFigure 5) In addition the orientation ofmotion in each blockis readily observable even by inspection in Figure 6(b) by aline separating the yellow part and blue part in each blockFurther results about the direction of motion are presentedin Section 4
34 Relationship to Biological Motion Detectors A straight-forward way to implement local motion detectors is to applya complex-valued Gabor receptive field (5) to the video signal119906 and then take the derivative of the phase with respect totime or apply a high-pass filter on the phase to approximatethe derivative
We present here an alternate implementation withoutexplicitly computing the phaseThiswill elucidate the relationbetween the phase-based motion detector presented in theprevious section and some elementary motion detectionmodels used in biology such as theReichardtmotion detector[7] and motion energy detector [9]
Assuming that the local phase 12060100(120596119909 120596119910 119905) is differen-
tiable we have (see also the Appendix)
11988912060100(120596119909 120596119910 119905)
119889119905
=
(119889119887 (120596119909 120596119910 119905) 119889119905) 119886 (120596
119909 120596119910 119905) minus (119889119886 (120596
119909 120596119910 119905) 119889119905) 119887 (120596
119909 120596119910 119905)
[119886 (120596119909 120596119910 119905)]
2
+ [119887 (120596119909 120596119910 119905)]
2 (42)
where 119886(120596119909 120596119910 119905) and 119887(120596
119909 120596119910 119905) are respectively the real
and imaginary parts of 11986000(120596119909 120596119910 119905)11989011989512060100(120596119909120596119910119905)
We notice that the denominator of (42) is the square ofthe local amplitude of 119906 and the numerator is of the formof a second order Volterra kernel This suggests that thetime derivative of the local phase can be viewed as a secondorder Volterra kernel that processes two normalized spatiallyfiltered inputs V
1and V2
We consider an elaborated Reichardt motion detector asshown in Figure 7 It is equipped with a quadrature pair ofGabor filters whose outputs are 119903
1(119905) = 119886(120596
119909 120596119910 119905) and
1199032(119905) = 119887(120596
119909 120596119910 119905) respectively for a particular value of
(120596119909 120596119910) The pair of Gabor filters that provide these outputs
are the real and imaginary parts of119908(119909 119910)119890minus119895(120596119909119909+120596119910119910) It also
consists of a temporal high-pass 1198921(119905) and temporal low-
pass filter 1198922(119905) [9] The output of the elaborated Reichardt
detector follows the diagram in Figure 7 and can be expressedas
(1199032lowast 1198921) (119905) (119903
1lowast 1198922) (119905) minus (119903
1lowast 1198921) (119905) (119903
2lowast 1198922) (119905) (43)
The response can also be characterized by a second orderVolterra kernel We notice the striking similarity between
(43) and the numerator of (42) In fact the phase-basedmotion detector shares some properties with the Reichardtmotion detector For example it is straightforward to see thata single phase-basedmotion detector is tuned to the temporalfrequency of a moving sinusoidal grating
Since the motion energy detector is formally equivalentto an elaborated Reichardt motion detector [9] the structureof the motion energy detector with divisive normalization isalso similar to the phase-based motion detector
4 Exploratory Results
In this section we apply the phase-based motion detectionalgorithm on several video sequences and demonstrate itsefficiency and effectiveness in detecting local motion Themotion detection algorithm is compared to two well-knownbiological motion detectors namely the Reichardt motiondetector and the Barlow-Levick motion detector We thenshow that the detected local motion can be used in motionsegmentation tasks The effectiveness of the segmentationis compared to segmentation using motion informationobtained from a widely used optic flow based algorithmavailable in the literature [4]
Computational Intelligence and Neuroscience 11
(f) (e)
(b)
(a)
(d)
(c)
minus120587 minus1205872 0 1205872
10080604020
y (pi
xel)
100 15050x (pixel)
0020406081
15
10
5
l
01234567+
15
10
5
l
0510152025+
minus6
minus4
minus2
0
2
4
6
181716151413121110
9876543210
l
2 4 6 8 10 12 14 16 18 20 22 240k
minus6minus4minus20246
minus6minus4minus20246
120596x
minus120587 minus1205872 0 1205872
120596x
1205872
0
minus1205872
minus120587
120596y
1205872
0
minus1205872
minus120587
120596y
10 15 20 255k
10 20 255 15k
Figure 6 Evaluating the motion detection algorithm at time 1199050(see also Supplementary Video S1) (a) A still from a 156 times 112-pixel video at
time 1199050 (b) (119889120601
119896119897119889119905)(120596
119909 120596119910 1199050) for all 26times19 blocks each of size 32times32The blocks are concatenated to create a large ldquophase imagerdquo with their
relative neighbors kept (c) (1198891206011211
119889119905)(120596119909 120596119910 1199050) shown in the (120596
119909 120596119910) isin D2 space (d) (119889120601
236119889119905)(120596
119909 120596119910 1199050) shown in the (120596
119909 120596119910) isin D2
space (e) Phase Motion Indicator (PMI) for all blocks at 1199050 Each pixel (119896 119897) is color coded for PMI
119896119897 (f) max
(120596119909 120596119910)isinD2 |(119889120601119896119897119889119905)(120596
119909 120596119910 1199050)|
for all blocks at 1199050 Each pixel (119896 119897) is color coded for this maximum value of block (119896 119897)
g1(t)g2(t) g1(t) g2(t)
times times
minus++
SpatialGaborfilter
SpatialGaborfilter
r1(t) r2(t)
u(x y t)
Figure 7 Diagram of an elaborated Reichardt motion detector Itconsists of a quadrature pair of Gabor filters whose outputs 119903
1and 1199032
provide inputs to the high-pass filters 1198921(119905) and the low-pass filters
1198922(119905)
Table 1 Processing Speeds of the proposed algorithm the Reichardtmotion detector and the Barlow-Levick Motion detector
Screen size Processing capability (frames per second)Proposed Reichardt Barlow-Levick
320 times 240 420 790 800720 times 576 135 685 6901280 times 720 60 274 2931920 times 1080 27 191 194
41 Efficient Parallel Implementation The algorithm wasimplemented in PyCUDA [23] and tested on an NVIDIAGeForce GTX TITAN GPU All computations use singleprecision floating points The processing speeds of the algo-rithm for several screen sizes are listed in Table 1 Clearlythe proposed phase-based motion detection algorithm hasreal-time capability to process video even with full HighDefinition screen size
For comparison we implemented the Reichardt motiondetector and the Barlow-Levick motion detector Theirrespective diagrams are shown in Figure 8 Note that we
12 Computational Intelligence and Neuroscience
ukl(t) uk+1l(t)
g1(t)
g2(t)
g1(t)
g2(t)
+ minus+
timestimes
(a)
minus
minus minus
+
ukl(t) uk+1l(t)
g1(t)g1(t)
g2(t) g2(t)
++
+
(b)
Figure 8 (a) A Reichardt motion detector detecting motion between the two points 119906119896119897and 119906
119896+1119897in the blurred and downsampled video
(b) A Barlow-Levick motion detector detecting motion between the two points 119906119896119897and 119906
119896+1119897in the blurred and downsampled video 119892
1(119905)
denotes a high-pass filter and 1198922(119905) denotes a low-pass filter
moved the high-pass filter ℎ1(119905) to the front of the low-pass fil-
ter ℎ2(119905) This configuration provides a superior performance
to the one in Figure 7 For both the Reichardt and the Barlow-Levick detectors the videos are first blurred by a Gaussianfilter (with the same variance as the Gaussian window in (30)used for the phase-based motion detector) and subsampledat the center of each overlapping block in the phase-basedmotion detector The subsampled video then provides inputsto two 2D arrays of the circuits shown in Figure 8 one forthe horizontal direction and one for the vertical directionThe outputs of the horizontal and vertical motion circuitsform an array of motion vectors that indicate the strengthand direction of motion The three tested motion detectorshave the same number of outputs as a result The processingspeeds of the Reichardt motion detector and the Barlow-Levick motion detector both implemented in PyCUDA andtested on the same GPU are shown in Table 1
Note that the Reichardt motion detector and the Barlow-Levick motion detector are highly efficient due to the sim-plicity of their algorithmsThe phase-basedmotion detectionalgorithm however is a much more sophisticated algorithmand yet it can be implemented in real-time using parallelcomputing devices
The fast GPU implementation is based on the FFTand Matrix-Matrix multiplication It is expected that thoseoperations can be efficiently implemented in hardware forexample FPGA
42 Examples of Phased-BasedMotionDetection Weappliedour motion detection algorithm on video sequences of theChange Detection 2014 dataset [22] that did not exhibitcamera egomotion For these video sequences the standarddeviation of the Gaussian window functions was set to 4
pixels and the block size was chosen to be 32 times 32 pixelsThreshold and normalization parameters were kept the samewith the exception of the ldquothermal videordquo (in order to dealwith larger background noise levels see below) We alsotested the same video sequences using the Reichardt motiondetector and the Barlow-Levick motion detector For theReichardt detector the high-pass filters were chosen as firstorder filters with a time constant of 200 milliseconds and thelow-pass filters were chosen as first order filters with a timeconstant of 300 milliseconds (assuming that the frame rate is50 frames per second)Thresholdwas set to 2 For the Barlow-Levick motion detector the time constant of the first high-pass filters was set to 250 milliseconds The low-pass filterswere the same as in the Reichardt detectorThreshold was setto 2
The first video was taken from a highway surveillancecamera (ldquohighway videordquo) under good illumination con-ditions and high contrast The video had moderate noiseparticularly on the road surface The detected motion isshown in the top left panel of Figure 9 (see SupplementaryVideo S2 for full video) The phase-based motion detectionalgorithm captured both the moving cars and the tree leavesmoving (due to the wind) In the time interval between the9th and 10th second the camera was slightly moved left-and right-wards within 5 frames again possibly due to thewind Movement due to this shift was captured by themotiondetection algorithm and the algorithm was fast enough tocorrectly determine the direction of this movement In thisvideo we already noted that this algorithm suffers from theaperture problem For example in front of the van where along horizontal edge is present the detectedmotion ismostlypointing downwards In addition to moving downwards theedge is alsomoving to the left howeverThis is expected since
Computational Intelligence and Neuroscience 13
(a) (b) (c)
Figure 9 Top motion detection of the ldquohighway videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) Red arrows indicate detected motion and its direction Bottom the contrast of the video was artificiallyreduced by 5-fold and the mean was reduced to 35 of the original The red arrows are duplicated from the motion detection result on theoriginal video as in top Blue arrows are the result of motion detection on the video with reduced contrast If motion is detected both in theoriginal video and in the video with reduced contrast then the arrow is shown in magenta (as a mix of blue and red) (see SupplementaryVideo S2 for full video)
the algorithm only detects motion locally and does not takeinto account the overall shape of any object
For comparison the motion detection results for theReichardt motion detector and the Barlow-Levick motiondetector are shown in the top middle and top right panelof Figure 9 (see Supplementary Video S2 for full video) TheReichardt detector performed relatively well when vehiclesare moving faster but the direction it predicts for vehiclesmoving slower for example on the back of the image is notaccurate In addition motion is still detected in some partsof the screen where the vehicles have just passed by Thedetection result for the Barlow-Levick motion detector waspoorer In particular the response to OFF edge movement isalways the opposite to the actual movement direction
We then squeezed the range of the screen intensity from[0 1] to [02 04] resulting in a video with a lower meanluminance and lower contrast The motion detection resultson the low-contrast video are shown in the bottom 3 panelsof Figure 9 (see Supplementary Video S2 for full video) Forreference the motion detection results for the original videoare shown in red arrows Motion detected in the low-contrastvideo is shown in blue arrows if no motion is detected forthe block in the original video If motion is detected in boththe original and the low-contrast video the arrows are shownin magenta Figure 9 clearly shows that while the motiondetection performance is degraded for all three detectors thephase-basedmotion detector performed still quite well in thelow-contrast video detecting most of the moving vehicles
The other two detectors missed many of the blocks wheremotion was detected in the original movie
To quantify how well each detector works under differentcontrast conditions we computed the ratio of unthresholdedoutput values of eachmotion detector between lower contrastand full contrast video For example in the case of phase-basedmotion detector we computed for each block the ratiobetween the PMI index for the lower contrast video and thatfor the full contrast videoThe ratios are then averaged acrossall blocks where motion is detected in the full contrast videoThis average for different contrasts is shown in Figure 10 bythe blue curve In the ideal case when the normalizationconstant 120598 is 0 the phase detector should produce invariantresponses to videos with different contrast The curve shownhere is mainly due to a nonzero 120598The ratios for the Reichardtmotion detector and the Barlow-Levick motion detectorare computed similarly and are shown in red and yellowrespectively It is clear that the phase-based motion detectorhas a superior performance across a range of contrast valuesAt 20 contrast the phase-based detector still has 50 ofthe PMI index value for full contrast video As expectedthe response of Barlow-Levick motion detector is linear withrespect to contrast and the Reichardt motion detector has aquadratic relation to contrast For a fixed threshold value alarger ratio equates to amore consistent performance at lowercontrast
The second video was captured by a surveillance camerain a train station (ldquotrain station videordquo) The video was
14 Computational Intelligence and Neuroscience
Phase-basedReichardtBarlow-Levick
0
02
04
06
08
1
Ratio
40 60 80200 100Contrast ()
Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined
under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm
The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)
The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression
With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)
43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient
objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area
For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold
We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually
We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm
We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)
These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm
5 Discussion
Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes
Computational Intelligence and Neuroscience 15
(a) (b) (c)
Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)
(a) (b) (c)
Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)
by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to
models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]
We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in
16 Computational Intelligence and Neuroscience
(a) (b) (c)
Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)
Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)
phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in
Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)
modeling in vivo biological vision systems This is consistentwith other recent findings [15]
Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure
Computational Intelligence and Neuroscience 17
(a) (b)
Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)
(a) (b)
Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)
(a) (b)
Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)
Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase
itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third
18 Computational Intelligence and Neuroscience
(a) (b)
Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)
instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection
The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions
Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors
We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well
We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It
can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes
Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems
Appendix
The Derivative of the Local Phase
Let
119886 (120596119909 120596119910 119905)
= int
R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596
119909119909 + 120596119910119910) 119889119909 119889119910
119887 (120596119909 120596119910 119905)
= minusint
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
(A1)
and therefore
11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)
= 119886 (120596119909 120596119910 119905) + 119895119887 (120596
119909 120596119910 119905)
(A2)
The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596
119909
120596119910 119905) 119886(120596
119909 120596119910 119905)) and therefore
11988912060100
119889119905
=
119886 (119889119887119889119905) minus 119887 (119889119886119889119905)
[119886]2+ [119887]2
(A3)
Computational Intelligence and Neuroscience 19
Now
119889119886
119889119905
(120596119909 120596119910 119905)
= int
R2
120597119906
120597119905
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
(A4)
For a purely translational signal with instantaneous spatialshift given by (119904
119909(119905) 119904119910(119905)) within the support of the window
we have
119889119886
119889119905
(120596119909 120596119910 119905)
= minusint
R2
119889119904119909
119889119905
(119905)
120597119906
120597119909
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus int
R2
119889119904119910
119889119905
(119905)
120597119906
120597119910
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119909
119889119904119909
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119910
119889119904119910
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199033=1199033(120596119909120596119910119905)
+ 120596119909
119889119904119909
119889119905
119887
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199034=1199034(120596119909120596119910119905)
+ 120596119910
119889119904119910
119889119905
119887
(A5)
Similarly
119889119887
119889119905
(120596119909 120596119910 119905) = minus
119889119904119909
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199035=1199035(120596119909120596119910119905)
minus 120596119909
119889119904119909
119889119905
119886 minus
119889119904119910
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199036=1199036(120596119909120596119910119905)
minus 120596119910
119889119904119910
119889119905
119886
(A6)
Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to
11988912060100
119889119905
(120596119909 120596119910 119905) =
minus (119889119904119909119889119905) [119903
3119887 + 1199035119886 + 120596119909 [
119886]2+ 120596119909 [
119887]2] minus (119889119904
119910119889119905) [119903
4119887 + 1199036119886 + 120596119910 [
119886]2+ 120596119910 [
119887]2]
[119886]2+ [119887]2
= minus
119889119904119909
119889119905
120596119909minus
119889119904119910
119889119905
120596119910minus
119889119904119909
119889119905
sdot
1199033119887 + 1199035119886
[119886]2+ [119887]2minus
119889119904119910
119889119905
sdot
1199034119887 + 1199036119886
[119886]2+ [119887]2
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
v00(120596119909120596119910119905)
(A7)
Disclosure
The authorrsquos names are alphabetically listed
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgment
The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232
References
[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010
[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010
[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014
[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010
20 Computational Intelligence and Neuroscience
[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013
[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014
[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956
[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010
[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985
[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965
[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981
[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015
[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014
[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999
[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015
[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001
[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996
[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980
[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013
[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013
[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014
[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014
[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach
to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012
[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990
[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987
[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998
Submit your manuscripts athttpwwwhindawicom
Computer Games Technology
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Distributed Sensor Networks
International Journal of
Advances in
FuzzySystems
Hindawi Publishing Corporationhttpwwwhindawicom
Volume 2014
International Journal of
ReconfigurableComputing
Hindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Applied Computational Intelligence and Soft Computing
thinspAdvancesthinspinthinsp
Artificial Intelligence
HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014
Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Journal of
Computer Networks and Communications
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation
httpwwwhindawicom Volume 2014
Advances in
Multimedia
International Journal of
Biomedical Imaging
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
ArtificialNeural Systems
Advances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience
Industrial EngineeringJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Human-ComputerInteraction
Advances in
Computer EngineeringAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience 3
where (120596119909 120596119910) isin R is the amplitude and
120601(120596119909 120596119910) is the
phase of the Fourier transform of 119906
Definition 1 The amplitude of the Fourier transform of animage 119906 = 119906(119909 119910) (119909 119910) isin R2 is called the global amplitudeof 119906 The phase of the Fourier transform of an image 119906 =
119906(119909 119910) (119909 119910) isin R2 is called the global phase of 119906
It is known that the global phase of an image plays animportant role in the representation of natural images [11] Aclassic example is to take two images and to exchange theirglobal phases before their reconstruction using the inverseFourier transform The resulting images are slightly smearedbut largely reflect the information contained in the globalphase
22The Local Phase of Images The global phase indicates theoffset of sinusoids of different frequencies contained in theentire image However it is not intuitive to relate the globalphase to local image features such as edges and their positionin an image To study these local features it is necessary tomodify the Fourier transform such that it reflects propertiesof a restricted region of an image It is natural to consider theShort-Time (-Space) Fourier Transform (STFT)
119880(120596119909 120596119910 1199090 1199100) = int
R2119906 (119909 119910)119908 (119909 minus 119909
0 119910 minus 119910
0)
sdot 119890minus119895(120596119909(119909minus1199090)+120596119910(119910minus1199100))119889119909 119889119910
(3)
where 119908 = 119908(119909 119910) (119909 119910) isin R2 is a real valued windowfunction centered at (119909
0 1199100) isin R2 Typical choices of window
functions include the Hann window and the Gaussian win-dow The (effectively) finite support of the window restrictsthe Fourier transform to local image analysis
Similarly to the Fourier transform the STFT can beexpressed in polar coordinates as
119880(120596119909 120596119910 1199090 1199100) = 119860 (120596
119909 120596119910 1199090 1199100) 119890119895120601(12059611990912059611991011990901199100) (4)
where 119860(120596119909 120596119910 1199090 1199100) isin R is the amplitude and 120601(120596
119909 120596119910
1199090 1199100) is the phase of the STFT
Definition 2 The amplitude of the STFT of an image 119906 =
119906(119909 119910) (119909 119910) isin R2 is called the local amplitude of 119906 Thephase of the STFT of an image 119906 = 119906(119909 119910) (119909 119910) isin R2 iscalled the local phase of 119906
Note that when 119908 is a Gaussian window the STFT of 119906evaluated at (120596
119909 120596119910 1199090 1199100) can be equivalently viewed as the
response of a complex-valued Gabor receptive field
ℎ (119909 119910) = 119890minus((119909minus119909
0)2+(119910minus119910
0)2)21205902
119890minus119895(120596119909(119909minus1199090)+120596119910(119910minus1199100)) (5)
to 119906In this case the window of the STFT clearly is given by
119908 (119909 119910) = 119890minus(1199092+1199102)21205902
(6)
Therefore the STFT can be realized by an ensemble of Gaborreceptive fields that are common in modeling simple andcomplex cells (neurons) in the primary visual cortex [16]
23 Reconstruction of Images from Local Phase Amplitudeand phase can be interpreted as measurementsprojectionsof images that are indicative of their information contentClassically when both the global amplitude and phase areknown it is straightforward to reconstruct the image Thereconstruction calls for computing the inverse Fourier trans-form given the global amplitude and phase Similarly whenusing local amplitude and phase if the sampling functionsform a basis or frame in a space of images the reconstructionis provided by the formalism of wavelet theory for exampleusing Gabor wavelets [17]
Amplitude or phase represents partial informationextracted from visual scenesThey are obtained via nonlinearsampling that is a nonlinear operation for extracting theamplitude andphase information from imagesThenonlinearoperation makes reconstruction from either amplitude orphase alone difficult Earlier studies and recent developmentin solving quadratic constraints however suggest that it ispossible to reconstruct images from global or local amplitudeinformation [18 19]
While computing the amplitude requires a second order(quadratic) nonlinearity computing the phase calls for higherorder nonlinear operators (Volterra kernels) It is possiblehowever to reconstruct up to a constant scale an image fromits global phase information alone without explicitly usingthe amplitude information An algorithm was provided forsolving this problem in the discrete signal processing domainin [11]The algorithm smartly avoids using the inverse tangentfunction by reformulating the phase measurements as a set oflinear equations that are easy to solve
Using a similar argument we demonstrate in the follow-ing that up to a constant scale a bandlimited signal 119906 =
119906(119909 119910) (119909 119910) isin R2 can be reconstructed from its local phasealone We first formulate the encoding of an image 119906 by localphase using the Gabor receptive fields as a special case Itis straightforward then to formulate the problem with localphase computed from other types of STFTs
Formally we consider an image 119906 = 119906(119909 119910) on thedomain R2 to be an element of a space of trigonometricpolynomialsH of the form
119906 (119909 119910) =
119871119909
sum
119897119909=minus119871119909
119871119910
sum
119897119910=minus119871119910
119888119897119909119897119910
119890119897119909119897119910
(119909 119910) (7)
where
119890119897119909119897119910
= exp(119895119897119909
Ω119909
119871119909
119909) exp(119895119897119910
Ω119910
119871119910
119910)
119897119909= minus119871119909 119871
119909 119897119910= minus119871119910 119871
119910
(8)
are the set of basis functions ofHΩ119909 119871119909are the bandwidth
and the order respectively of the space in the 119909 dimensionand Ω
119910 119871119910are the bandwidth and the order respectively of
the space in the 119910 dimensionH is a Reproducing Kernel Hilbert Space with inner
product [20 21]
⟨1199061 1199062⟩ = int
[0119879119909]times[0119879
119910]
1199061(119909 119910) 119906
2(119909 119910)119889119909 119889119910 (9)
4 Computational Intelligence and Neuroscience
where 119879119909= 2120587119871
119909Ω119909 119879119910= 2120587119871
119910Ω119910are the period of the
space in the 119909 and 119910 dimensions respectivelyConsider a bank of119873 Gabor receptive fields
ℎ119896119897119898119899
(119909 119910) = T119896119897(119908 (119909 119910) 119890
minus119895(120596119909119898119909+120596119910119899119910)) (10)
where T119896119897is the translation operator with T
119896119897119906 = 119906(119909 minus
1198961198870 119910 minus 119897119887
0) 119896 119897 isin Z 119887
0gt 0 0 le 119896119887
0le 119879119909 and 0 le
1198971198870le 119879119910 and 120596
119909119898
= 1198981205960 120596119910119899
= 1198991205960 119898 119899 isin Z 120596
0gt 0
minusΩ119909le 119898120596
0le Ω119909 and minusΩ
119910le 1198991205960le Ω119910 The responses of
the Gabor receptive fields to the input 119906(119909 119910) are given by
int
R2119906 (119909 119910) ℎ
119896119897119898119899(119909 119910) 119889119909 119889119910 = int
R2119906 (119909 119910)
sdot 119908 (119909 minus 1198961198870 119910 minus 119897119887
0) 119890minus119895(120596119909119898(119909minus119896119887
0)+120596119910119899(119910minus1198971198870))119889119909 119889119910
= 119860119896119897119898119899
119890119895120601119896119897119898119899
(11)
where 119860119896119897119898119899
ge 0 119860119896119897119898119899
isin R is the local amplitude and120601119896119897119898119899
isin [0 2120587) is the local phaseDividing both sides of (11) by 119890119895120601119896119897119898119899 we have
int
R2119906 (119909 119910)119908 (119909 minus 119896119887
0 119910 minus 119897119887
0)
sdot 119890minus119895(120596119909119898(119909minus119896119887
0)+120596119910119899(119910minus1198971198870)+120601119896119897119898119899)119889119909 119889119910 = 119860
119896119897119898119899
(12)
Since 119860119896119897119898119899
isin R we have
int
R2119906 (119909 119910)119908 (119909 minus 119896119887
0 119910 minus 119897119887
0)
sdot cos (120596119909119898
(119909 minus 1198961198870) + 120596119910119899
(119910 minus 1198971198870) + 120601119896119897119898119899
) 119889119909 119889119910
= 119860119896119897119898119899
(13)
int
R2119906 (119909 119910)119908 (119909 minus 119896119887
0 119910 minus 119897119887
0)
sdot sin (120596119909119898
(119909 minus 1198961198870) + 120596119910119899
(119910 minus 1198971198870) + 120601119896119897119898119899
) 119889119909 119889119910
= 0
(14)
Note that119908(119909 minus 1198961198870 119910 minus 119897119887
0) sin(120596
119909119898
(119909 minus 1198961198870) + 120596119910119899
(119910 minus 1198971198870) +
120601119896119897119898119899
) is a real-valued Gabor receptive field with a preferredphase at 120601
119896119897119898119899+ 1205872 minus 120596
119909119898
1198961198870minus 120596119910119899
1198971198870
Remark 3 Assuming that the local phase information 120601119896119897119898119899
is obtained via measurements that is filtering the image 119906
with pairs of Gabor receptive fields (10) the set of linearequations (14) has a simple interpretation the image 119906 isorthogonal to the space spanned by the functions
119908 (119909 minus 1198961198870 119910 minus 119897119887
0)
sdot sin (120596119909119898
(119909 minus 1198961198870) + 120596119910119899
(119910 minus 1198971198870) + 120601119896119897119898119899
)
(15)
where (119896 119897 119898 119899) isin I with I = (119896 119897 119898 119899) isin Z4 | 0 le 1198961198870le
119879119909 0 le 119897119887
0le 119879119910 minusΩ119909le 119898120596
0le Ω119909 minusΩ119910le 1198991205960le Ω119910
We are now in the position to provide a reconstructionalgorithm of the image from phase 120601
119896119897119898119899 (119896 119897 119898 119899) isin I
Lemma 4 119906 can be reconstructed from 120601119896119897119898119899
(119896 119897 119898 119899) isin Ias
119906 (119909 119910) =
119871119909
sum
119897119909=minus119871119909
119871119910
sum
119897119910=minus119871119910
119888119897119909119897119910
119890119897119909119897119910
(119909 119910) (16)
with
Φ119888 = 0 (17)
whereΦ is a matrix whose 119901th row and 119902th column entry are
[Φ]119901119902
= int
R2119908 (119909 minus 119896119887
0 119910 minus 119897119887
0)
sdot sin (120596119909119898
(119909 minus 1198961198870) + 120596119910119899
(119910 minus 1198971198870) + 120601119896119897119898119899
)
sdot 119890119897119909119897119910
(119909 119910) 119889119909 119889119910
(18)
Here 119901 traverses the set I and 119902 = (2119871119910+ 1)(119897119909+ 119871119909) + (119897119910+
119871119910+ 1) 119888 is a vector of the form
119888 = [119888minus119871119909minus119871119910
119888minus119871119909minus119871119910+1 119888
minus119871119909119871119910
119888minus119871119909+1minus119871
119910
119888minus119871119909+1minus119871
119910+1 119888
minus119871119909+1119871119910
119888119871119909minus119871119910
119888119871119909minus119871119910+1
119888119871119909119871119910
]
119879
(19)
that belongs to the null space of Φ A necessary conditionfor perfect reconstruction of 119906 up to a constant scale is that119873 ge (2119871
119909+ 1)(2119871
119910+ 1) minus 1 where 119873 is the number of phase
measurements
Proof Substituting (7) into (14) we obtain119871119909
sum
119897119909=minus119871119909
119871119910
sum
119897119910=minus119871119910
119888119897119909119897119910
int
R2119908 (119909 minus 119896119887
0 119910 minus 119897119887
0)
sdot sin (120596119909119898
(119909 minus 1198961198870) + 120596119910119899
(119910 minus 1198971198870) + 120601119896119897119898119899
)
sdot 119890119897119909119897119910
(119909 119910) 119889119909 119889119910 = 0
(20)
for all 119896 119897 119898 119899 isin Z Therefore we have
Φ119888 = 0 (21)
inferring that 119888 is in the null space ofΦIf 119873 lt (2119871
119909+ 1)(2119871
119910+ 1) minus 1 it follows from the rank-
nullity theorem that dim(null(Φ)) gt 1 leading to multiplelinearly independent solutions toΦ119888 = 0
Example 5 In Figure 2 an example of reconstruction of animage is shown using only local phase information Thereconstructed signal was scaled to match the original signalThe SNR of the reconstruction is 4448 [dB] An alternativeway to obtain a unique reconstruction is to include anadditional measurement for example the mean value of thesignal int
R2119906(119909 119910)119889119909 119889119910 to the system of linear equations (14)
Computational Intelligence and Neuroscience 5
(a) (b) (c)
Figure 2 An example of reconstruction of image from local phase information (a) Original image the dimension of the space is 16129(b) Reconstructed image from 20102 phase measurements scaled to match the original (c) Error
3 Visual Motion Detection fromPhase Information
In this section we consider visual fields that change as afunction of time For notational simplicity 119906 will denote herethe space-time intensity of the visual field
31 The Global Phase Equation for Translational Motion Let119906 = 119906(119909 119910 119905) (119909 119910) isin R2 119905 isin R be a visual stimulus If thevisual stimulus is a pure translation of the signal at 119906(119909 119910 0)that is
119906 (119909 119910 119905) = 119906 (119909 minus 119904119909(119905) 119910 minus 119904
119910(119905) 0) (22)
where
119904119909(119905) = int
119905
0
V119909(119904) 119889119904
119904119910(119905) = int
119905
0
V119910(119904) 119889119904
(23)
are the total length of translation at time 119905 in each dimen-sion and V
119909(119905) and V
119910(119905) are the corresponding instanta-
neous velocity components then the only difference between119906(119909 119910 119905) and 119906(119909 119910 0) in the Fourier domain is captured bytheir global phase More formally consider the following
Lemma 6 The change (derivative) of the global phase is givenby
119889120601 (120596119909 120596119910 119905)
119889119905
= minus120596119909V119909(119905) minus 120596
119910V119910(119905)
= minus [120596119909 120596119910] [V119909(119905) V119910(119905)]
119879
(24)
where by abuse of notation 120601(120596119909 120596119910 119905) denotes the global
phase of 119906(119909 119910 119905) and 120601(120596119909 120596119910 0) is the initial condition
Proof If (F119906(sdot sdot 0)) (120596119909 120596119910) is the 2D (spatial) Fourier
transform of 119906 = 119906(119909 119910 0) (119909 119910) isin R2 by the Fourier shifttheorem we have
(F119906 (sdot sdot 119905)) (120596119909 120596119910)
= (F119906 (sdot sdot 0)) (120596119909 120596119910) 119890minus119895(120596119909119904119909(119905)+120596
119910119904119910(119905))
(25)
For a certain frequency component (120596119909 120596119910) the change of its
global phase over time amounts to
119889120601 (120596119909 120596119910 119905)
119889119905
= minus120596119909
119889119904119909(119905)
119889119905
minus 120596119910
119889119904119910(119905)
119889119905
= minus120596119909V119909(119905) minus 120596
119910V119910(119905)
= minus [120596119909 120596119910] [V119909(119905) V119910(119905)]
119879
(26)
Therefore in the simple case where the entire visual field isshifting the derivative of the phase of Fourier componentsindicates motion and it can be obtained by the innerproduct between the component frequency and the velocityvector
32 The Change of Local Phase
321The Local Phase Equation for Translational Motion Theanalysis in Section 31 applies to global motion This type ofmotion occurs most frequently when the imaging deviceeither an eye or a cameramoves Visualmotion in the naturalenvironment however ismore diverse across the screen sinceit is often time produced by multiple moving objects Theobjects can be small and the motion more localized in thevisual field
Taking the global phase of 119906 will not simply revealwhere motion of independent objects takes place or theirdirectionvelocity of motion The ease of interpretation ofmotion by using the Fourier transform however motivates
6 Computational Intelligence and Neuroscience
us to reuse the same concept in detecting local motion Thiscan be achieved by restricting the domain of the visual fieldwhere the Fourier transform is applied
To be able to detect local motion we consider thelocal phase of 119906(119909 119910 119905) by taking the STFT with windowfunction 119908(119909 119910) Note that the STFT and its ubiquitousimplementation in DSP chips can be extensively used in anydimension For simplicity and without loss of generality weconsider the window to be centered at (0 0) The STFT isgiven by
int
R2119906 (119909 119910 119905) 119908 (119909 119910) 119890
minus119895(120596119909119909+120596119910119910)119889119909 119889119910
= 11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)
(27)
where by abuse of notation 11986000(120596119909 120596119910 119905) is the amplitude
and 12060100(120596119909 120596119910 119905) the local phase
Before we move on to the mathematical analysis we canintuitively explain the relation between the change in localphase and visual motion taking place across the windowsupport First if the stimulus undergoes a uniform changeof intensity or it changes proportionally over time due tolighting conditions for example the local phase does notchange since the phase is invariant with respect to intensityscaling Therefore the local phase does not change for suchnonmotion stimuli Second a rigid edge moving across thewindow support will induce a phase change
For a strictly translational signal within the windowsupport (footprint) for example
119906 (119909 119910 119905) = 119906 (119909 minus 119904119909(119905) 119910 minus 119904
119910(119905) 0)
for (119909 119910) isin supp (119908 (119909 119910))
(28)
where 119904119909(119905) and 119904
119910(119905) are as defined in (23) we have the
following result
Lemma 7 Consider the following
11988912060100
119889119905
(120596119909 120596119910 119905) = minus
119889119904119909(119905)
119889119905
120596119909minus
119889119904119910(119905)
119889119905
120596119910
+ v00(120596119909 120596119910 119905)
(29)
where by abuse of notation 12060100(120596119909 120596119910 119905) is the local phase
of 119906(119909 119910 119905) and 12060100(120596119909 120596119910 0) is the initial condition The
functional form of the term v00
= v00(120596119909 120596119910 119905) is provided
in the Appendix
Proof The derivation of (29) and the functional form of v00
are given in the Appendix
Remark 8 Wenotice that the derivative of the local phase hassimilar structure to that of the global phase but for the addedterm v
00Through simulations we observed that the first two
terms in (29) dominate over the last term for an ON or OFFmoving edge [3] For example Figure 3 shows the derivativeof the local phase given in (29) for an ON edge moving withvelocity (40 0) pixelssec
42
0minus2
minus4
100
d120601dt
120596yminus250minus200minus150minus100minus50
050
150
200250
31 2minus4 0minus1minus3 minus2 4
120596x
Figure 3 Derivative of the local phase for an ON edge moving inthe positive 119909 direction
Remark 9 Note that 12060100(120596119909 120596119910 119905) may not be differen-
tiable even if 119906(119909 119910 119905) is differentiable particularly when11986000(120596119909 120596119910 119905) = 0 For example the spatial phase can jump
from a positive value to zero when11986000(120596119909 120596119910 119905) diminishes
This also suggests that the instantaneous local spatial phaseis less informative about a region of a visual scene whenever11986000(120596119909 120596119910 119905) is close to zero Nevertheless the time deriva-
tive of the local phase can be approximated by applying ahigh-pass filter to 120601
00(120596119909 120596119910 119905)
322 The Block Structure for Computing the Local Phase Weconstruct Gaussian windows along the 119909 119910 dimensions TheGaussian windows are defined as
(T119896119897119908) (119909 119910) = 119890
minus((119909minus119909119896)2+(119909minus119910
119897)2)21205902
(30)
where 119909119896= 1198961198870 119910119897= 1198971198870 in which 119887
0isin Z+ is the distance
between two neighboring windows and 1 le 1198961198870le 119875119909 1 le
1198971198870le 119875119910 where 119875
119909 119875119910isin Z+ are the number of pixels of the
screen in 119909 and 119910 directions respectivelyWe then take the 2D Fourier transform of the windowed
video signal 119906(119909 119910 119905)(T119896119897119908)(119909 119910) and write in polar form
int
R2119906 (119909 119910 119905) (T
119896119897119908) (119909 119910) 119890
minus119895(120596119909(119909minus119909119896)+120596119910(119910minus119910119897))119889119909 119889119910
= 119860119896119897(120596119909 120596119910 119905) 119890119895120601119896119897(120596119909120596119910119905)
(31)
The above integral can be very efficiently evaluated usingthe 2D FFT in discrete domain defined on 119872 times 119872 blocksapproximating the footprint of the Gaussian windows Forexample the standard deviation of the Gaussian windows weuse in the examples in Section 4 is 4 pixels A block of 32times32
pixels (119872 = 32) is sufficient to cover the effective support(or footprint) of the Gaussian window At the same time thesize of the block is a power of 2 which is most suitable forFFT-based computationThe processing of each block (119896 119897) isindependent of all other blocks thereby parallelism is readilyachieved
Note that the size of the window is informed by thesize of the objects one is interested in locating Measure-ments of the local phase using smaller window functions
Computational Intelligence and Neuroscience 7
60
50
40
30
20
10
y (pi
xel)
01
02
03
04
05
06
07
08
09
1
20 30 40 50 6010x (pixel)
(a)
0
02
04
06
08
1
Mag
nitu
de (a
u)
10 20 30 40 50 60 700y (pixel)
(b)
Figure 4 Example of block structure (a) Four yellow disks show 4 Gaussian window functions with translation parameter 119896 = 3 119897 = 3
(top-left) 119896 = 3 119897 = 7 (bottom-left) 119896 = 7 119897 = 3 (top-right) and 119896 = 7 119897 = 7 (bottom-right) The red solid square shows a 32 times 32-pixel blockapproximating the Gaussian window with 119896 = 3 119897 = 3 and the green dashed square shows a 32 times 32-pixel block approximating the Gaussianwindow with 119896 = 3 119897 = 7 (b) Cross section of all 11 Gaussian windows with translation parameters 119896 = 7 119897 isin [0 10] The cross section istaken as indicated by the magenta line in (a) and the red and blue curve correspond to cross sections of the two Gaussian windows shown in(a) centered on the magenta line
are less robust to noise Larger windows would enhanceobject motion detection if the object size is comparable tothe window size However there would be an increasedlikelihood of independent movement of multiple objectswithin the same window which is not modeled here andthereby may not be robustly detected
Therefore for each block (119896 119897) we obtain 1198722 measure-
ments of the phase 120601119896119897(120596119909119898
120596119910119898
119905) at every time instant 119905with (120596
119909119898
120596119910119898
) isin D2 where
D2= (120596
119909119898
= 1198981205960 120596119910119899
= 1198991205960) 119898 119899 = minus1198722
minus 1198722 + 1 1198722 minus 1
(32)
with 1205960= 2120587119872 We then compute the temporal derivative
of the phase that is (119889120601119896119897119889119905)(120596
119909 120596119910 119905) for (120596
119909 120596119910) isin D2
We further illustrate an example of the block structurein Figure 4 Figure 4(a) shows an example of an image of64 times 64 pixels Four Gaussian windows are shown each witha standard deviation of 4 pixels The distance between thecenters of two neighboring Gaussian windows is 6 pixelsThe red solid square shows a 32 times 32-pixel block with 119896 =
3 119897 = 3 which encloses effective support of the Gaussianwindow on top-left (119896 = 0 119897 = 0 is the block with Gaussianwindow centered at pixel (1 1)) The green dashed squareshows another 32 times 32-pixel block with 119896 = 3 119897 = 7 Thetwo Gaussian windows on the right are associated with theblocks 119896 = 7 119897 = 3 and 119896 = 7 119897 = 7 respectively Crosssection of all Gaussian windows with 119896 = 7 119897 isin [0 10] that isthose centered on themagenta line are shown in Figure 4(b)The red and blue curve in Figure 4(b) correspond to the two
Gaussian windows shown in Figure 4(a) Figure 4(b) alsosuggests that some of the Gaussian windows are cut off on theboundaries This is however equivalent to assuming that thepixel values outside the boundary are always zero and it willnot significantly affect motion detection based on the changeof local phase
Since the phase and thereby the phase change is noisierwhen the local amplitude is low an additional denoisingstep can be employed to discount the measurements of(119889120601119896119897119889119905)(120596
119909 120596119910 119905) for low amplitude values 119860
119896119897(120596119909 120596119910 119905)
The denoising is given by
119889120601119896119897
119889119905
(120596119909 120596119910 119905)
sdot
119860119896119897(120596119909 120596119910 119905)
(11198722)sum(120596119909120596119910)isinD2 119860119896119897 (120596119909 120596119910 119905) + 120598
(33)
where 120598 gt 0 is a constant and (120596119909 120596119910) isin D2
33 The Phase-Based Detector We propose here a block FFTbased algorithm to detect motion using phase informationSuch an algorithm is due to its simplicity and parallelismhighly suitable for an in silico implementation
331 Radon Transform on the Change of Phases We exploitthe approximately linear structure of the phase derivative forblocks exhibiting motion by computing the Radon transformof (119889120601
119896119897119889119905)(120596
119909 120596119910 119905) over a circular bounded domain 119862 =
(120596119909 120596119910) | (120596119909 120596119910) isin D2 1205962
119909+ 1205962
119910lt 1205872
8 Computational Intelligence and Neuroscience
The Radon transform of the change of phase in thedomain 119862 is given by
(R119889120601119896119897
119889119905
) (120588 120579 119905) = int
R
119889120601119896119897
119889119905
(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588
sdot sin 120579 + 119904 sdot cos 120579 119905) 1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579
+ 119904 sdot cos 120579) 119889119904
(34)
where
1119862(120596119909 120596119910) =
1 if (120596119909 120596119910) isin 119862
0 otherwise(35)
The Radon transform (R(119889120601119896119897119889119905))(120588 120579 119905) evaluated at
a particular point (1205880 1205790 1199050) is essentially an integral
of (119889120601119896119897119889119905)(120596
119909 120596119910 1199050) along a line oriented at angle
1205872 + 1205790with the 120596
119909axis and at distance |120588
0| along the
(cos(1205790) sin(120579
0)) direction from (0 0)
If for a particular 119896 and 119897 we have (119889120601119896119897119889119905)(120596
119909 120596119910 119905) =
minusV119909(119905)120596119909minus V119910(119905)120596119910 we have
(R (119889120601119896119897119889119905)) (120588 120579 119905)
c (120588 120579)
= 120588 [minusV119909(119905) cos 120579 minus V
119910(119905) sin 120579]
(36)
where
c (120588 120579)
= int
R
1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579 + 119904 sdot cos 120579) 119889119904
(37)
c(120588 120579) is a correction term due to different length of lineintegrals for different values of (120588 120579) in the bounded domain119862
After computing the Radon transform of (119889120601119896119897119889119905)(120596
119909
120596119910 119905) for every block (119896 119897) at time 119905
0 we compute the Phase
Motion Indicator (PMI) defined as
PMI119896119897= max120579isin[0120587)
sum
120588
100381610038161003816100381610038161003816100381610038161003816
(R (119889120601119896119897119889119905)) (120588 120579 119905
0)
c (120588 120579)
100381610038161003816100381610038161003816100381610038161003816
(38)
If the PMI119896119897
is larger than a chosen threshold motion isdeemed to occur in block (119896 119897) at time 119905
0
Using the Radon transform makes it easier to separaterigid motion from noise Since the phase is quite sensitiveto noise particularly when the amplitude is very smallthe change of phase under noise may have comparablemagnitude to that due to motion as mentioned earlier Thechange of phase under noise however does not possess thestructure suggested by (29) in the (120596
119909 120596119910) domain Instead
it appears to be more randomly distributed Consequently
the PMI value is comparatively small for these blocks (see alsoSection 333)
Moreover the direction of motion for block (119896 119897) wheremotion is detected can be easily computed as
120579119896119897
= 120587(
sign (sum120588gt0
(R (119889120601119896119897119889119905)) (120588 120572
119896119897 1199050) c (120588 120572
119896119897)) + 1
2
)
+ 120572119896119897
(39)
where
120572119896119897= argmax120579isin[0120587)
sum
120588
100381610038161003816100381610038161003816100381610038161003816
(R (119889120601119896119897119889119905)) (120588 120579 119905
0)
c (120588 120579)
100381610038161003816100381610038161003816100381610038161003816
(40)
This follows from (36)
332 The Phase-Based Motion Detection Algorithm We for-mally summarize the above analysis as Algorithm 1 Figure 5shows a schematic diagram of the proposed phase-basedmotion detection algorithm
The algorithm is subdivided into two parts The first partcomputes local phase changes and the second part is thephase-based motion detector
In the first part the screen is divided into overlappingblocks For example the red green and blue blocks in theplane ldquodivide into overlapping blocksrdquo correspond to thesquares of the same color covering the video stream AGaussian window is then applied on each block followed bya 2D FFT operation that is used to extract the local phase Atemporal high-pass filter is then employed to extract phasechanges
In the second part the PMI is evaluated for each blockbased on the Radon transform of the local phase changes ineach block Motion is detected for blocks with PMI largerthan a preset threshold and the direction of motion iscomputed as in (39)
It is easy to notice that the algorithm can be highlyparallelized
333 Example We provide an illustrative example inFigure 6 showing how motion is detected using Algorithm 1The full video of this example can be found in SupplementaryVideo S1 see Supplementary Material available online athttpdxdoiorg10115520167915245 Figure 6(a) depicts astill from the ldquohighway videordquo in the Change Detection 2014dataset [22] evaluated at a particular time 119905
0 As suggested
by the algorithm the screen in Figure 6(a) is divided into26 times 19 overlapping blocks and the window functions areapplied to each block Local phases can then be extractedfrom the 2DFFT of eachwindowed block and the local phasechanges are obtained by temporal high-pass filtering Thephase change is shown in Figure 6(b) for all blocks with block(12 11) enlarged in Figure 6(c) and block (23 6) enlarged inFigure 6(d) (see also the plane ldquo2D FFT and extract phasechangerdquo in Figure 5) Note that at the time of the videoframe block (12 11) covers a part of the vehicle in motion
Computational Intelligence and Neuroscience 9
Input Visual stream or Video data 119906(119909 119910 119905)Output Detected MotionConstruct Gaussian window of size119872times119872 denote as 119908for 119905 = Δ119879 2Δ119879 119873Δ119879 do
Divide the screen of frame at time 119905 into multiple119872times119872 blocks 119906119896119897(119905) with overlaps
foreach block 119906119896119897(119905) do
Multiply pixel-wise the block 119906119896119897(119905) with the Gaussian window 119906
119896119897(119905)119908
Take119872times119872 2D FFT of 119906119896119897(119905)119908 denote phase as 120601
119896119897(120596119909 120596119910 119905) and amplitude as 119860
119896119897(120596119909 120596119910 119905)
foreach frequency (120596119909 120596119910) do
Highpass filter temporally 120601119896119897(120596119909 120596119910 119905) denote 1206011015840
119896119897(120596119909 120596119910 119905)
Optionally denoise by 1206011015840119896119897(120596119909 120596119910 119905) larr 120601
1015840
119896119897(120596119909 120596119910 119905)(119860
119896119897(120596119909 120596119910 119905)((1119872
2) sum(120596119909 120596119910)isinD
2 119860119896119897(120596119909 120596119910 119905) + 120598))
endCompute the radon transform (R120601
1015840
119896119897)(120588 120579 119905)
Compute PMI119896119897(119905) according to (38)
if PMI119896119897(119905) gt threshold then
Motion Detected at block (119896 119897) at time 119905Compute direction of motion according to (39)
endend
end
Algorithm 1 Phase-based motion detection algorithm using the FFT
y
120596x
120596y
y
t
120596x
120596y
x
y
x
y
x
x
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
Divided intooverlapping blocks
Apply window function
2D FFT andextract phase change
Radon transformand extract strength
and orientation of plane
Thresholding
Computephase
change
The phase-baseddetector
Figure 5 Schematic diagram of proposed phase-based motion detection algorithm
10 Computational Intelligence and Neuroscience
in the front and block (23 6) corresponds to an area of thehighway pavement where no motion occurs
Figure 6(f) depicts for each block (119896 119897) the maximumphase change over all (120596
119909 120596119910) isin D2 that is
max(120596119909120596119910)isinD2
10038161003816100381610038161003816100381610038161003816
119889120601119896119897
119889119905
(120596119909 120596119910 1199050)
10038161003816100381610038161003816100381610038161003816
(41)
We observe from the figure that for regions with lowamplitude such as the region depicting the road whenthe normalization constant is absent the derivative of thephase can be noisy For these blocks the maximum of|(119889120601119896119897119889119905)(120596
119909 120596119910 1199050)| over all (120596
119909 120596119910) isin D2 is comparable
to the maximum obtained for blocks that cover the vehiclesin motion
However (29) suggests that the local phase change frommultiple filter pairs centered at the same spatial position(119896 119897) can provide a constraint to robustly estimate motionand its direction Given the block structure employed in thecomputation of the local phase it is natural to utilize phasechange information from multiple sources
Indeed if for a particular block (119896 119897) (119889120601119896119897119889119905)(120596
119909
120596119910 119905) = minusV
119909(119905)120596119909minus V119910(119905)120596119910 then it is easy to see that (119889120601
119896119897
119889119905)(120596119909 120596119910 119905) will be zero on the line V
119909(119905)120596119909+ V119910(119905)120596119910= 0
and have opposite sign on either side of this line For examplein Figures 6(b) and 6(c) (119889120601
119896119897119889119905)(120596
119909 120596119910 1199050) clearly exhibits
this property for blocks that cover a vehicle in motion ThePMI is a tool to evaluate this property
Finally the PMIs for all blocks are shown compactlyin a heat map in Figure 6(e) The figure shows clearly thatthe blocks corresponding to the two moving vehicles have ahigh PMI value while the stationary background areas havea low PMI value allowing one to easily detect motion byemploying simple thresholding (see also the plane ldquoRadontransform and extract strength orientation of planerdquo inFigure 5) In addition the orientation ofmotion in each blockis readily observable even by inspection in Figure 6(b) by aline separating the yellow part and blue part in each blockFurther results about the direction of motion are presentedin Section 4
34 Relationship to Biological Motion Detectors A straight-forward way to implement local motion detectors is to applya complex-valued Gabor receptive field (5) to the video signal119906 and then take the derivative of the phase with respect totime or apply a high-pass filter on the phase to approximatethe derivative
We present here an alternate implementation withoutexplicitly computing the phaseThiswill elucidate the relationbetween the phase-based motion detector presented in theprevious section and some elementary motion detectionmodels used in biology such as theReichardtmotion detector[7] and motion energy detector [9]
Assuming that the local phase 12060100(120596119909 120596119910 119905) is differen-
tiable we have (see also the Appendix)
11988912060100(120596119909 120596119910 119905)
119889119905
=
(119889119887 (120596119909 120596119910 119905) 119889119905) 119886 (120596
119909 120596119910 119905) minus (119889119886 (120596
119909 120596119910 119905) 119889119905) 119887 (120596
119909 120596119910 119905)
[119886 (120596119909 120596119910 119905)]
2
+ [119887 (120596119909 120596119910 119905)]
2 (42)
where 119886(120596119909 120596119910 119905) and 119887(120596
119909 120596119910 119905) are respectively the real
and imaginary parts of 11986000(120596119909 120596119910 119905)11989011989512060100(120596119909120596119910119905)
We notice that the denominator of (42) is the square ofthe local amplitude of 119906 and the numerator is of the formof a second order Volterra kernel This suggests that thetime derivative of the local phase can be viewed as a secondorder Volterra kernel that processes two normalized spatiallyfiltered inputs V
1and V2
We consider an elaborated Reichardt motion detector asshown in Figure 7 It is equipped with a quadrature pair ofGabor filters whose outputs are 119903
1(119905) = 119886(120596
119909 120596119910 119905) and
1199032(119905) = 119887(120596
119909 120596119910 119905) respectively for a particular value of
(120596119909 120596119910) The pair of Gabor filters that provide these outputs
are the real and imaginary parts of119908(119909 119910)119890minus119895(120596119909119909+120596119910119910) It also
consists of a temporal high-pass 1198921(119905) and temporal low-
pass filter 1198922(119905) [9] The output of the elaborated Reichardt
detector follows the diagram in Figure 7 and can be expressedas
(1199032lowast 1198921) (119905) (119903
1lowast 1198922) (119905) minus (119903
1lowast 1198921) (119905) (119903
2lowast 1198922) (119905) (43)
The response can also be characterized by a second orderVolterra kernel We notice the striking similarity between
(43) and the numerator of (42) In fact the phase-basedmotion detector shares some properties with the Reichardtmotion detector For example it is straightforward to see thata single phase-basedmotion detector is tuned to the temporalfrequency of a moving sinusoidal grating
Since the motion energy detector is formally equivalentto an elaborated Reichardt motion detector [9] the structureof the motion energy detector with divisive normalization isalso similar to the phase-based motion detector
4 Exploratory Results
In this section we apply the phase-based motion detectionalgorithm on several video sequences and demonstrate itsefficiency and effectiveness in detecting local motion Themotion detection algorithm is compared to two well-knownbiological motion detectors namely the Reichardt motiondetector and the Barlow-Levick motion detector We thenshow that the detected local motion can be used in motionsegmentation tasks The effectiveness of the segmentationis compared to segmentation using motion informationobtained from a widely used optic flow based algorithmavailable in the literature [4]
Computational Intelligence and Neuroscience 11
(f) (e)
(b)
(a)
(d)
(c)
minus120587 minus1205872 0 1205872
10080604020
y (pi
xel)
100 15050x (pixel)
0020406081
15
10
5
l
01234567+
15
10
5
l
0510152025+
minus6
minus4
minus2
0
2
4
6
181716151413121110
9876543210
l
2 4 6 8 10 12 14 16 18 20 22 240k
minus6minus4minus20246
minus6minus4minus20246
120596x
minus120587 minus1205872 0 1205872
120596x
1205872
0
minus1205872
minus120587
120596y
1205872
0
minus1205872
minus120587
120596y
10 15 20 255k
10 20 255 15k
Figure 6 Evaluating the motion detection algorithm at time 1199050(see also Supplementary Video S1) (a) A still from a 156 times 112-pixel video at
time 1199050 (b) (119889120601
119896119897119889119905)(120596
119909 120596119910 1199050) for all 26times19 blocks each of size 32times32The blocks are concatenated to create a large ldquophase imagerdquo with their
relative neighbors kept (c) (1198891206011211
119889119905)(120596119909 120596119910 1199050) shown in the (120596
119909 120596119910) isin D2 space (d) (119889120601
236119889119905)(120596
119909 120596119910 1199050) shown in the (120596
119909 120596119910) isin D2
space (e) Phase Motion Indicator (PMI) for all blocks at 1199050 Each pixel (119896 119897) is color coded for PMI
119896119897 (f) max
(120596119909 120596119910)isinD2 |(119889120601119896119897119889119905)(120596
119909 120596119910 1199050)|
for all blocks at 1199050 Each pixel (119896 119897) is color coded for this maximum value of block (119896 119897)
g1(t)g2(t) g1(t) g2(t)
times times
minus++
SpatialGaborfilter
SpatialGaborfilter
r1(t) r2(t)
u(x y t)
Figure 7 Diagram of an elaborated Reichardt motion detector Itconsists of a quadrature pair of Gabor filters whose outputs 119903
1and 1199032
provide inputs to the high-pass filters 1198921(119905) and the low-pass filters
1198922(119905)
Table 1 Processing Speeds of the proposed algorithm the Reichardtmotion detector and the Barlow-Levick Motion detector
Screen size Processing capability (frames per second)Proposed Reichardt Barlow-Levick
320 times 240 420 790 800720 times 576 135 685 6901280 times 720 60 274 2931920 times 1080 27 191 194
41 Efficient Parallel Implementation The algorithm wasimplemented in PyCUDA [23] and tested on an NVIDIAGeForce GTX TITAN GPU All computations use singleprecision floating points The processing speeds of the algo-rithm for several screen sizes are listed in Table 1 Clearlythe proposed phase-based motion detection algorithm hasreal-time capability to process video even with full HighDefinition screen size
For comparison we implemented the Reichardt motiondetector and the Barlow-Levick motion detector Theirrespective diagrams are shown in Figure 8 Note that we
12 Computational Intelligence and Neuroscience
ukl(t) uk+1l(t)
g1(t)
g2(t)
g1(t)
g2(t)
+ minus+
timestimes
(a)
minus
minus minus
+
ukl(t) uk+1l(t)
g1(t)g1(t)
g2(t) g2(t)
++
+
(b)
Figure 8 (a) A Reichardt motion detector detecting motion between the two points 119906119896119897and 119906
119896+1119897in the blurred and downsampled video
(b) A Barlow-Levick motion detector detecting motion between the two points 119906119896119897and 119906
119896+1119897in the blurred and downsampled video 119892
1(119905)
denotes a high-pass filter and 1198922(119905) denotes a low-pass filter
moved the high-pass filter ℎ1(119905) to the front of the low-pass fil-
ter ℎ2(119905) This configuration provides a superior performance
to the one in Figure 7 For both the Reichardt and the Barlow-Levick detectors the videos are first blurred by a Gaussianfilter (with the same variance as the Gaussian window in (30)used for the phase-based motion detector) and subsampledat the center of each overlapping block in the phase-basedmotion detector The subsampled video then provides inputsto two 2D arrays of the circuits shown in Figure 8 one forthe horizontal direction and one for the vertical directionThe outputs of the horizontal and vertical motion circuitsform an array of motion vectors that indicate the strengthand direction of motion The three tested motion detectorshave the same number of outputs as a result The processingspeeds of the Reichardt motion detector and the Barlow-Levick motion detector both implemented in PyCUDA andtested on the same GPU are shown in Table 1
Note that the Reichardt motion detector and the Barlow-Levick motion detector are highly efficient due to the sim-plicity of their algorithmsThe phase-basedmotion detectionalgorithm however is a much more sophisticated algorithmand yet it can be implemented in real-time using parallelcomputing devices
The fast GPU implementation is based on the FFTand Matrix-Matrix multiplication It is expected that thoseoperations can be efficiently implemented in hardware forexample FPGA
42 Examples of Phased-BasedMotionDetection Weappliedour motion detection algorithm on video sequences of theChange Detection 2014 dataset [22] that did not exhibitcamera egomotion For these video sequences the standarddeviation of the Gaussian window functions was set to 4
pixels and the block size was chosen to be 32 times 32 pixelsThreshold and normalization parameters were kept the samewith the exception of the ldquothermal videordquo (in order to dealwith larger background noise levels see below) We alsotested the same video sequences using the Reichardt motiondetector and the Barlow-Levick motion detector For theReichardt detector the high-pass filters were chosen as firstorder filters with a time constant of 200 milliseconds and thelow-pass filters were chosen as first order filters with a timeconstant of 300 milliseconds (assuming that the frame rate is50 frames per second)Thresholdwas set to 2 For the Barlow-Levick motion detector the time constant of the first high-pass filters was set to 250 milliseconds The low-pass filterswere the same as in the Reichardt detectorThreshold was setto 2
The first video was taken from a highway surveillancecamera (ldquohighway videordquo) under good illumination con-ditions and high contrast The video had moderate noiseparticularly on the road surface The detected motion isshown in the top left panel of Figure 9 (see SupplementaryVideo S2 for full video) The phase-based motion detectionalgorithm captured both the moving cars and the tree leavesmoving (due to the wind) In the time interval between the9th and 10th second the camera was slightly moved left-and right-wards within 5 frames again possibly due to thewind Movement due to this shift was captured by themotiondetection algorithm and the algorithm was fast enough tocorrectly determine the direction of this movement In thisvideo we already noted that this algorithm suffers from theaperture problem For example in front of the van where along horizontal edge is present the detectedmotion ismostlypointing downwards In addition to moving downwards theedge is alsomoving to the left howeverThis is expected since
Computational Intelligence and Neuroscience 13
(a) (b) (c)
Figure 9 Top motion detection of the ldquohighway videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) Red arrows indicate detected motion and its direction Bottom the contrast of the video was artificiallyreduced by 5-fold and the mean was reduced to 35 of the original The red arrows are duplicated from the motion detection result on theoriginal video as in top Blue arrows are the result of motion detection on the video with reduced contrast If motion is detected both in theoriginal video and in the video with reduced contrast then the arrow is shown in magenta (as a mix of blue and red) (see SupplementaryVideo S2 for full video)
the algorithm only detects motion locally and does not takeinto account the overall shape of any object
For comparison the motion detection results for theReichardt motion detector and the Barlow-Levick motiondetector are shown in the top middle and top right panelof Figure 9 (see Supplementary Video S2 for full video) TheReichardt detector performed relatively well when vehiclesare moving faster but the direction it predicts for vehiclesmoving slower for example on the back of the image is notaccurate In addition motion is still detected in some partsof the screen where the vehicles have just passed by Thedetection result for the Barlow-Levick motion detector waspoorer In particular the response to OFF edge movement isalways the opposite to the actual movement direction
We then squeezed the range of the screen intensity from[0 1] to [02 04] resulting in a video with a lower meanluminance and lower contrast The motion detection resultson the low-contrast video are shown in the bottom 3 panelsof Figure 9 (see Supplementary Video S2 for full video) Forreference the motion detection results for the original videoare shown in red arrows Motion detected in the low-contrastvideo is shown in blue arrows if no motion is detected forthe block in the original video If motion is detected in boththe original and the low-contrast video the arrows are shownin magenta Figure 9 clearly shows that while the motiondetection performance is degraded for all three detectors thephase-basedmotion detector performed still quite well in thelow-contrast video detecting most of the moving vehicles
The other two detectors missed many of the blocks wheremotion was detected in the original movie
To quantify how well each detector works under differentcontrast conditions we computed the ratio of unthresholdedoutput values of eachmotion detector between lower contrastand full contrast video For example in the case of phase-basedmotion detector we computed for each block the ratiobetween the PMI index for the lower contrast video and thatfor the full contrast videoThe ratios are then averaged acrossall blocks where motion is detected in the full contrast videoThis average for different contrasts is shown in Figure 10 bythe blue curve In the ideal case when the normalizationconstant 120598 is 0 the phase detector should produce invariantresponses to videos with different contrast The curve shownhere is mainly due to a nonzero 120598The ratios for the Reichardtmotion detector and the Barlow-Levick motion detectorare computed similarly and are shown in red and yellowrespectively It is clear that the phase-based motion detectorhas a superior performance across a range of contrast valuesAt 20 contrast the phase-based detector still has 50 ofthe PMI index value for full contrast video As expectedthe response of Barlow-Levick motion detector is linear withrespect to contrast and the Reichardt motion detector has aquadratic relation to contrast For a fixed threshold value alarger ratio equates to amore consistent performance at lowercontrast
The second video was captured by a surveillance camerain a train station (ldquotrain station videordquo) The video was
14 Computational Intelligence and Neuroscience
Phase-basedReichardtBarlow-Levick
0
02
04
06
08
1
Ratio
40 60 80200 100Contrast ()
Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined
under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm
The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)
The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression
With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)
43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient
objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area
For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold
We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually
We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm
We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)
These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm
5 Discussion
Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes
Computational Intelligence and Neuroscience 15
(a) (b) (c)
Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)
(a) (b) (c)
Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)
by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to
models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]
We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in
16 Computational Intelligence and Neuroscience
(a) (b) (c)
Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)
Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)
phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in
Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)
modeling in vivo biological vision systems This is consistentwith other recent findings [15]
Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure
Computational Intelligence and Neuroscience 17
(a) (b)
Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)
(a) (b)
Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)
(a) (b)
Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)
Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase
itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third
18 Computational Intelligence and Neuroscience
(a) (b)
Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)
instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection
The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions
Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors
We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well
We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It
can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes
Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems
Appendix
The Derivative of the Local Phase
Let
119886 (120596119909 120596119910 119905)
= int
R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596
119909119909 + 120596119910119910) 119889119909 119889119910
119887 (120596119909 120596119910 119905)
= minusint
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
(A1)
and therefore
11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)
= 119886 (120596119909 120596119910 119905) + 119895119887 (120596
119909 120596119910 119905)
(A2)
The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596
119909
120596119910 119905) 119886(120596
119909 120596119910 119905)) and therefore
11988912060100
119889119905
=
119886 (119889119887119889119905) minus 119887 (119889119886119889119905)
[119886]2+ [119887]2
(A3)
Computational Intelligence and Neuroscience 19
Now
119889119886
119889119905
(120596119909 120596119910 119905)
= int
R2
120597119906
120597119905
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
(A4)
For a purely translational signal with instantaneous spatialshift given by (119904
119909(119905) 119904119910(119905)) within the support of the window
we have
119889119886
119889119905
(120596119909 120596119910 119905)
= minusint
R2
119889119904119909
119889119905
(119905)
120597119906
120597119909
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus int
R2
119889119904119910
119889119905
(119905)
120597119906
120597119910
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119909
119889119904119909
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119910
119889119904119910
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199033=1199033(120596119909120596119910119905)
+ 120596119909
119889119904119909
119889119905
119887
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199034=1199034(120596119909120596119910119905)
+ 120596119910
119889119904119910
119889119905
119887
(A5)
Similarly
119889119887
119889119905
(120596119909 120596119910 119905) = minus
119889119904119909
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199035=1199035(120596119909120596119910119905)
minus 120596119909
119889119904119909
119889119905
119886 minus
119889119904119910
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199036=1199036(120596119909120596119910119905)
minus 120596119910
119889119904119910
119889119905
119886
(A6)
Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to
11988912060100
119889119905
(120596119909 120596119910 119905) =
minus (119889119904119909119889119905) [119903
3119887 + 1199035119886 + 120596119909 [
119886]2+ 120596119909 [
119887]2] minus (119889119904
119910119889119905) [119903
4119887 + 1199036119886 + 120596119910 [
119886]2+ 120596119910 [
119887]2]
[119886]2+ [119887]2
= minus
119889119904119909
119889119905
120596119909minus
119889119904119910
119889119905
120596119910minus
119889119904119909
119889119905
sdot
1199033119887 + 1199035119886
[119886]2+ [119887]2minus
119889119904119910
119889119905
sdot
1199034119887 + 1199036119886
[119886]2+ [119887]2
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
v00(120596119909120596119910119905)
(A7)
Disclosure
The authorrsquos names are alphabetically listed
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgment
The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232
References
[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010
[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010
[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014
[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010
20 Computational Intelligence and Neuroscience
[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013
[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014
[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956
[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010
[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985
[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965
[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981
[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015
[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014
[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999
[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015
[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001
[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996
[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980
[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013
[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013
[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014
[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014
[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach
to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012
[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990
[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987
[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998
Submit your manuscripts athttpwwwhindawicom
Computer Games Technology
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Distributed Sensor Networks
International Journal of
Advances in
FuzzySystems
Hindawi Publishing Corporationhttpwwwhindawicom
Volume 2014
International Journal of
ReconfigurableComputing
Hindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Applied Computational Intelligence and Soft Computing
thinspAdvancesthinspinthinsp
Artificial Intelligence
HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014
Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Journal of
Computer Networks and Communications
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation
httpwwwhindawicom Volume 2014
Advances in
Multimedia
International Journal of
Biomedical Imaging
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
ArtificialNeural Systems
Advances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience
Industrial EngineeringJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Human-ComputerInteraction
Advances in
Computer EngineeringAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
4 Computational Intelligence and Neuroscience
where 119879119909= 2120587119871
119909Ω119909 119879119910= 2120587119871
119910Ω119910are the period of the
space in the 119909 and 119910 dimensions respectivelyConsider a bank of119873 Gabor receptive fields
ℎ119896119897119898119899
(119909 119910) = T119896119897(119908 (119909 119910) 119890
minus119895(120596119909119898119909+120596119910119899119910)) (10)
where T119896119897is the translation operator with T
119896119897119906 = 119906(119909 minus
1198961198870 119910 minus 119897119887
0) 119896 119897 isin Z 119887
0gt 0 0 le 119896119887
0le 119879119909 and 0 le
1198971198870le 119879119910 and 120596
119909119898
= 1198981205960 120596119910119899
= 1198991205960 119898 119899 isin Z 120596
0gt 0
minusΩ119909le 119898120596
0le Ω119909 and minusΩ
119910le 1198991205960le Ω119910 The responses of
the Gabor receptive fields to the input 119906(119909 119910) are given by
int
R2119906 (119909 119910) ℎ
119896119897119898119899(119909 119910) 119889119909 119889119910 = int
R2119906 (119909 119910)
sdot 119908 (119909 minus 1198961198870 119910 minus 119897119887
0) 119890minus119895(120596119909119898(119909minus119896119887
0)+120596119910119899(119910minus1198971198870))119889119909 119889119910
= 119860119896119897119898119899
119890119895120601119896119897119898119899
(11)
where 119860119896119897119898119899
ge 0 119860119896119897119898119899
isin R is the local amplitude and120601119896119897119898119899
isin [0 2120587) is the local phaseDividing both sides of (11) by 119890119895120601119896119897119898119899 we have
int
R2119906 (119909 119910)119908 (119909 minus 119896119887
0 119910 minus 119897119887
0)
sdot 119890minus119895(120596119909119898(119909minus119896119887
0)+120596119910119899(119910minus1198971198870)+120601119896119897119898119899)119889119909 119889119910 = 119860
119896119897119898119899
(12)
Since 119860119896119897119898119899
isin R we have
int
R2119906 (119909 119910)119908 (119909 minus 119896119887
0 119910 minus 119897119887
0)
sdot cos (120596119909119898
(119909 minus 1198961198870) + 120596119910119899
(119910 minus 1198971198870) + 120601119896119897119898119899
) 119889119909 119889119910
= 119860119896119897119898119899
(13)
int
R2119906 (119909 119910)119908 (119909 minus 119896119887
0 119910 minus 119897119887
0)
sdot sin (120596119909119898
(119909 minus 1198961198870) + 120596119910119899
(119910 minus 1198971198870) + 120601119896119897119898119899
) 119889119909 119889119910
= 0
(14)
Note that119908(119909 minus 1198961198870 119910 minus 119897119887
0) sin(120596
119909119898
(119909 minus 1198961198870) + 120596119910119899
(119910 minus 1198971198870) +
120601119896119897119898119899
) is a real-valued Gabor receptive field with a preferredphase at 120601
119896119897119898119899+ 1205872 minus 120596
119909119898
1198961198870minus 120596119910119899
1198971198870
Remark 3 Assuming that the local phase information 120601119896119897119898119899
is obtained via measurements that is filtering the image 119906
with pairs of Gabor receptive fields (10) the set of linearequations (14) has a simple interpretation the image 119906 isorthogonal to the space spanned by the functions
119908 (119909 minus 1198961198870 119910 minus 119897119887
0)
sdot sin (120596119909119898
(119909 minus 1198961198870) + 120596119910119899
(119910 minus 1198971198870) + 120601119896119897119898119899
)
(15)
where (119896 119897 119898 119899) isin I with I = (119896 119897 119898 119899) isin Z4 | 0 le 1198961198870le
119879119909 0 le 119897119887
0le 119879119910 minusΩ119909le 119898120596
0le Ω119909 minusΩ119910le 1198991205960le Ω119910
We are now in the position to provide a reconstructionalgorithm of the image from phase 120601
119896119897119898119899 (119896 119897 119898 119899) isin I
Lemma 4 119906 can be reconstructed from 120601119896119897119898119899
(119896 119897 119898 119899) isin Ias
119906 (119909 119910) =
119871119909
sum
119897119909=minus119871119909
119871119910
sum
119897119910=minus119871119910
119888119897119909119897119910
119890119897119909119897119910
(119909 119910) (16)
with
Φ119888 = 0 (17)
whereΦ is a matrix whose 119901th row and 119902th column entry are
[Φ]119901119902
= int
R2119908 (119909 minus 119896119887
0 119910 minus 119897119887
0)
sdot sin (120596119909119898
(119909 minus 1198961198870) + 120596119910119899
(119910 minus 1198971198870) + 120601119896119897119898119899
)
sdot 119890119897119909119897119910
(119909 119910) 119889119909 119889119910
(18)
Here 119901 traverses the set I and 119902 = (2119871119910+ 1)(119897119909+ 119871119909) + (119897119910+
119871119910+ 1) 119888 is a vector of the form
119888 = [119888minus119871119909minus119871119910
119888minus119871119909minus119871119910+1 119888
minus119871119909119871119910
119888minus119871119909+1minus119871
119910
119888minus119871119909+1minus119871
119910+1 119888
minus119871119909+1119871119910
119888119871119909minus119871119910
119888119871119909minus119871119910+1
119888119871119909119871119910
]
119879
(19)
that belongs to the null space of Φ A necessary conditionfor perfect reconstruction of 119906 up to a constant scale is that119873 ge (2119871
119909+ 1)(2119871
119910+ 1) minus 1 where 119873 is the number of phase
measurements
Proof Substituting (7) into (14) we obtain119871119909
sum
119897119909=minus119871119909
119871119910
sum
119897119910=minus119871119910
119888119897119909119897119910
int
R2119908 (119909 minus 119896119887
0 119910 minus 119897119887
0)
sdot sin (120596119909119898
(119909 minus 1198961198870) + 120596119910119899
(119910 minus 1198971198870) + 120601119896119897119898119899
)
sdot 119890119897119909119897119910
(119909 119910) 119889119909 119889119910 = 0
(20)
for all 119896 119897 119898 119899 isin Z Therefore we have
Φ119888 = 0 (21)
inferring that 119888 is in the null space ofΦIf 119873 lt (2119871
119909+ 1)(2119871
119910+ 1) minus 1 it follows from the rank-
nullity theorem that dim(null(Φ)) gt 1 leading to multiplelinearly independent solutions toΦ119888 = 0
Example 5 In Figure 2 an example of reconstruction of animage is shown using only local phase information Thereconstructed signal was scaled to match the original signalThe SNR of the reconstruction is 4448 [dB] An alternativeway to obtain a unique reconstruction is to include anadditional measurement for example the mean value of thesignal int
R2119906(119909 119910)119889119909 119889119910 to the system of linear equations (14)
Computational Intelligence and Neuroscience 5
(a) (b) (c)
Figure 2 An example of reconstruction of image from local phase information (a) Original image the dimension of the space is 16129(b) Reconstructed image from 20102 phase measurements scaled to match the original (c) Error
3 Visual Motion Detection fromPhase Information
In this section we consider visual fields that change as afunction of time For notational simplicity 119906 will denote herethe space-time intensity of the visual field
31 The Global Phase Equation for Translational Motion Let119906 = 119906(119909 119910 119905) (119909 119910) isin R2 119905 isin R be a visual stimulus If thevisual stimulus is a pure translation of the signal at 119906(119909 119910 0)that is
119906 (119909 119910 119905) = 119906 (119909 minus 119904119909(119905) 119910 minus 119904
119910(119905) 0) (22)
where
119904119909(119905) = int
119905
0
V119909(119904) 119889119904
119904119910(119905) = int
119905
0
V119910(119904) 119889119904
(23)
are the total length of translation at time 119905 in each dimen-sion and V
119909(119905) and V
119910(119905) are the corresponding instanta-
neous velocity components then the only difference between119906(119909 119910 119905) and 119906(119909 119910 0) in the Fourier domain is captured bytheir global phase More formally consider the following
Lemma 6 The change (derivative) of the global phase is givenby
119889120601 (120596119909 120596119910 119905)
119889119905
= minus120596119909V119909(119905) minus 120596
119910V119910(119905)
= minus [120596119909 120596119910] [V119909(119905) V119910(119905)]
119879
(24)
where by abuse of notation 120601(120596119909 120596119910 119905) denotes the global
phase of 119906(119909 119910 119905) and 120601(120596119909 120596119910 0) is the initial condition
Proof If (F119906(sdot sdot 0)) (120596119909 120596119910) is the 2D (spatial) Fourier
transform of 119906 = 119906(119909 119910 0) (119909 119910) isin R2 by the Fourier shifttheorem we have
(F119906 (sdot sdot 119905)) (120596119909 120596119910)
= (F119906 (sdot sdot 0)) (120596119909 120596119910) 119890minus119895(120596119909119904119909(119905)+120596
119910119904119910(119905))
(25)
For a certain frequency component (120596119909 120596119910) the change of its
global phase over time amounts to
119889120601 (120596119909 120596119910 119905)
119889119905
= minus120596119909
119889119904119909(119905)
119889119905
minus 120596119910
119889119904119910(119905)
119889119905
= minus120596119909V119909(119905) minus 120596
119910V119910(119905)
= minus [120596119909 120596119910] [V119909(119905) V119910(119905)]
119879
(26)
Therefore in the simple case where the entire visual field isshifting the derivative of the phase of Fourier componentsindicates motion and it can be obtained by the innerproduct between the component frequency and the velocityvector
32 The Change of Local Phase
321The Local Phase Equation for Translational Motion Theanalysis in Section 31 applies to global motion This type ofmotion occurs most frequently when the imaging deviceeither an eye or a cameramoves Visualmotion in the naturalenvironment however ismore diverse across the screen sinceit is often time produced by multiple moving objects Theobjects can be small and the motion more localized in thevisual field
Taking the global phase of 119906 will not simply revealwhere motion of independent objects takes place or theirdirectionvelocity of motion The ease of interpretation ofmotion by using the Fourier transform however motivates
6 Computational Intelligence and Neuroscience
us to reuse the same concept in detecting local motion Thiscan be achieved by restricting the domain of the visual fieldwhere the Fourier transform is applied
To be able to detect local motion we consider thelocal phase of 119906(119909 119910 119905) by taking the STFT with windowfunction 119908(119909 119910) Note that the STFT and its ubiquitousimplementation in DSP chips can be extensively used in anydimension For simplicity and without loss of generality weconsider the window to be centered at (0 0) The STFT isgiven by
int
R2119906 (119909 119910 119905) 119908 (119909 119910) 119890
minus119895(120596119909119909+120596119910119910)119889119909 119889119910
= 11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)
(27)
where by abuse of notation 11986000(120596119909 120596119910 119905) is the amplitude
and 12060100(120596119909 120596119910 119905) the local phase
Before we move on to the mathematical analysis we canintuitively explain the relation between the change in localphase and visual motion taking place across the windowsupport First if the stimulus undergoes a uniform changeof intensity or it changes proportionally over time due tolighting conditions for example the local phase does notchange since the phase is invariant with respect to intensityscaling Therefore the local phase does not change for suchnonmotion stimuli Second a rigid edge moving across thewindow support will induce a phase change
For a strictly translational signal within the windowsupport (footprint) for example
119906 (119909 119910 119905) = 119906 (119909 minus 119904119909(119905) 119910 minus 119904
119910(119905) 0)
for (119909 119910) isin supp (119908 (119909 119910))
(28)
where 119904119909(119905) and 119904
119910(119905) are as defined in (23) we have the
following result
Lemma 7 Consider the following
11988912060100
119889119905
(120596119909 120596119910 119905) = minus
119889119904119909(119905)
119889119905
120596119909minus
119889119904119910(119905)
119889119905
120596119910
+ v00(120596119909 120596119910 119905)
(29)
where by abuse of notation 12060100(120596119909 120596119910 119905) is the local phase
of 119906(119909 119910 119905) and 12060100(120596119909 120596119910 0) is the initial condition The
functional form of the term v00
= v00(120596119909 120596119910 119905) is provided
in the Appendix
Proof The derivation of (29) and the functional form of v00
are given in the Appendix
Remark 8 Wenotice that the derivative of the local phase hassimilar structure to that of the global phase but for the addedterm v
00Through simulations we observed that the first two
terms in (29) dominate over the last term for an ON or OFFmoving edge [3] For example Figure 3 shows the derivativeof the local phase given in (29) for an ON edge moving withvelocity (40 0) pixelssec
42
0minus2
minus4
100
d120601dt
120596yminus250minus200minus150minus100minus50
050
150
200250
31 2minus4 0minus1minus3 minus2 4
120596x
Figure 3 Derivative of the local phase for an ON edge moving inthe positive 119909 direction
Remark 9 Note that 12060100(120596119909 120596119910 119905) may not be differen-
tiable even if 119906(119909 119910 119905) is differentiable particularly when11986000(120596119909 120596119910 119905) = 0 For example the spatial phase can jump
from a positive value to zero when11986000(120596119909 120596119910 119905) diminishes
This also suggests that the instantaneous local spatial phaseis less informative about a region of a visual scene whenever11986000(120596119909 120596119910 119905) is close to zero Nevertheless the time deriva-
tive of the local phase can be approximated by applying ahigh-pass filter to 120601
00(120596119909 120596119910 119905)
322 The Block Structure for Computing the Local Phase Weconstruct Gaussian windows along the 119909 119910 dimensions TheGaussian windows are defined as
(T119896119897119908) (119909 119910) = 119890
minus((119909minus119909119896)2+(119909minus119910
119897)2)21205902
(30)
where 119909119896= 1198961198870 119910119897= 1198971198870 in which 119887
0isin Z+ is the distance
between two neighboring windows and 1 le 1198961198870le 119875119909 1 le
1198971198870le 119875119910 where 119875
119909 119875119910isin Z+ are the number of pixels of the
screen in 119909 and 119910 directions respectivelyWe then take the 2D Fourier transform of the windowed
video signal 119906(119909 119910 119905)(T119896119897119908)(119909 119910) and write in polar form
int
R2119906 (119909 119910 119905) (T
119896119897119908) (119909 119910) 119890
minus119895(120596119909(119909minus119909119896)+120596119910(119910minus119910119897))119889119909 119889119910
= 119860119896119897(120596119909 120596119910 119905) 119890119895120601119896119897(120596119909120596119910119905)
(31)
The above integral can be very efficiently evaluated usingthe 2D FFT in discrete domain defined on 119872 times 119872 blocksapproximating the footprint of the Gaussian windows Forexample the standard deviation of the Gaussian windows weuse in the examples in Section 4 is 4 pixels A block of 32times32
pixels (119872 = 32) is sufficient to cover the effective support(or footprint) of the Gaussian window At the same time thesize of the block is a power of 2 which is most suitable forFFT-based computationThe processing of each block (119896 119897) isindependent of all other blocks thereby parallelism is readilyachieved
Note that the size of the window is informed by thesize of the objects one is interested in locating Measure-ments of the local phase using smaller window functions
Computational Intelligence and Neuroscience 7
60
50
40
30
20
10
y (pi
xel)
01
02
03
04
05
06
07
08
09
1
20 30 40 50 6010x (pixel)
(a)
0
02
04
06
08
1
Mag
nitu
de (a
u)
10 20 30 40 50 60 700y (pixel)
(b)
Figure 4 Example of block structure (a) Four yellow disks show 4 Gaussian window functions with translation parameter 119896 = 3 119897 = 3
(top-left) 119896 = 3 119897 = 7 (bottom-left) 119896 = 7 119897 = 3 (top-right) and 119896 = 7 119897 = 7 (bottom-right) The red solid square shows a 32 times 32-pixel blockapproximating the Gaussian window with 119896 = 3 119897 = 3 and the green dashed square shows a 32 times 32-pixel block approximating the Gaussianwindow with 119896 = 3 119897 = 7 (b) Cross section of all 11 Gaussian windows with translation parameters 119896 = 7 119897 isin [0 10] The cross section istaken as indicated by the magenta line in (a) and the red and blue curve correspond to cross sections of the two Gaussian windows shown in(a) centered on the magenta line
are less robust to noise Larger windows would enhanceobject motion detection if the object size is comparable tothe window size However there would be an increasedlikelihood of independent movement of multiple objectswithin the same window which is not modeled here andthereby may not be robustly detected
Therefore for each block (119896 119897) we obtain 1198722 measure-
ments of the phase 120601119896119897(120596119909119898
120596119910119898
119905) at every time instant 119905with (120596
119909119898
120596119910119898
) isin D2 where
D2= (120596
119909119898
= 1198981205960 120596119910119899
= 1198991205960) 119898 119899 = minus1198722
minus 1198722 + 1 1198722 minus 1
(32)
with 1205960= 2120587119872 We then compute the temporal derivative
of the phase that is (119889120601119896119897119889119905)(120596
119909 120596119910 119905) for (120596
119909 120596119910) isin D2
We further illustrate an example of the block structurein Figure 4 Figure 4(a) shows an example of an image of64 times 64 pixels Four Gaussian windows are shown each witha standard deviation of 4 pixels The distance between thecenters of two neighboring Gaussian windows is 6 pixelsThe red solid square shows a 32 times 32-pixel block with 119896 =
3 119897 = 3 which encloses effective support of the Gaussianwindow on top-left (119896 = 0 119897 = 0 is the block with Gaussianwindow centered at pixel (1 1)) The green dashed squareshows another 32 times 32-pixel block with 119896 = 3 119897 = 7 Thetwo Gaussian windows on the right are associated with theblocks 119896 = 7 119897 = 3 and 119896 = 7 119897 = 7 respectively Crosssection of all Gaussian windows with 119896 = 7 119897 isin [0 10] that isthose centered on themagenta line are shown in Figure 4(b)The red and blue curve in Figure 4(b) correspond to the two
Gaussian windows shown in Figure 4(a) Figure 4(b) alsosuggests that some of the Gaussian windows are cut off on theboundaries This is however equivalent to assuming that thepixel values outside the boundary are always zero and it willnot significantly affect motion detection based on the changeof local phase
Since the phase and thereby the phase change is noisierwhen the local amplitude is low an additional denoisingstep can be employed to discount the measurements of(119889120601119896119897119889119905)(120596
119909 120596119910 119905) for low amplitude values 119860
119896119897(120596119909 120596119910 119905)
The denoising is given by
119889120601119896119897
119889119905
(120596119909 120596119910 119905)
sdot
119860119896119897(120596119909 120596119910 119905)
(11198722)sum(120596119909120596119910)isinD2 119860119896119897 (120596119909 120596119910 119905) + 120598
(33)
where 120598 gt 0 is a constant and (120596119909 120596119910) isin D2
33 The Phase-Based Detector We propose here a block FFTbased algorithm to detect motion using phase informationSuch an algorithm is due to its simplicity and parallelismhighly suitable for an in silico implementation
331 Radon Transform on the Change of Phases We exploitthe approximately linear structure of the phase derivative forblocks exhibiting motion by computing the Radon transformof (119889120601
119896119897119889119905)(120596
119909 120596119910 119905) over a circular bounded domain 119862 =
(120596119909 120596119910) | (120596119909 120596119910) isin D2 1205962
119909+ 1205962
119910lt 1205872
8 Computational Intelligence and Neuroscience
The Radon transform of the change of phase in thedomain 119862 is given by
(R119889120601119896119897
119889119905
) (120588 120579 119905) = int
R
119889120601119896119897
119889119905
(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588
sdot sin 120579 + 119904 sdot cos 120579 119905) 1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579
+ 119904 sdot cos 120579) 119889119904
(34)
where
1119862(120596119909 120596119910) =
1 if (120596119909 120596119910) isin 119862
0 otherwise(35)
The Radon transform (R(119889120601119896119897119889119905))(120588 120579 119905) evaluated at
a particular point (1205880 1205790 1199050) is essentially an integral
of (119889120601119896119897119889119905)(120596
119909 120596119910 1199050) along a line oriented at angle
1205872 + 1205790with the 120596
119909axis and at distance |120588
0| along the
(cos(1205790) sin(120579
0)) direction from (0 0)
If for a particular 119896 and 119897 we have (119889120601119896119897119889119905)(120596
119909 120596119910 119905) =
minusV119909(119905)120596119909minus V119910(119905)120596119910 we have
(R (119889120601119896119897119889119905)) (120588 120579 119905)
c (120588 120579)
= 120588 [minusV119909(119905) cos 120579 minus V
119910(119905) sin 120579]
(36)
where
c (120588 120579)
= int
R
1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579 + 119904 sdot cos 120579) 119889119904
(37)
c(120588 120579) is a correction term due to different length of lineintegrals for different values of (120588 120579) in the bounded domain119862
After computing the Radon transform of (119889120601119896119897119889119905)(120596
119909
120596119910 119905) for every block (119896 119897) at time 119905
0 we compute the Phase
Motion Indicator (PMI) defined as
PMI119896119897= max120579isin[0120587)
sum
120588
100381610038161003816100381610038161003816100381610038161003816
(R (119889120601119896119897119889119905)) (120588 120579 119905
0)
c (120588 120579)
100381610038161003816100381610038161003816100381610038161003816
(38)
If the PMI119896119897
is larger than a chosen threshold motion isdeemed to occur in block (119896 119897) at time 119905
0
Using the Radon transform makes it easier to separaterigid motion from noise Since the phase is quite sensitiveto noise particularly when the amplitude is very smallthe change of phase under noise may have comparablemagnitude to that due to motion as mentioned earlier Thechange of phase under noise however does not possess thestructure suggested by (29) in the (120596
119909 120596119910) domain Instead
it appears to be more randomly distributed Consequently
the PMI value is comparatively small for these blocks (see alsoSection 333)
Moreover the direction of motion for block (119896 119897) wheremotion is detected can be easily computed as
120579119896119897
= 120587(
sign (sum120588gt0
(R (119889120601119896119897119889119905)) (120588 120572
119896119897 1199050) c (120588 120572
119896119897)) + 1
2
)
+ 120572119896119897
(39)
where
120572119896119897= argmax120579isin[0120587)
sum
120588
100381610038161003816100381610038161003816100381610038161003816
(R (119889120601119896119897119889119905)) (120588 120579 119905
0)
c (120588 120579)
100381610038161003816100381610038161003816100381610038161003816
(40)
This follows from (36)
332 The Phase-Based Motion Detection Algorithm We for-mally summarize the above analysis as Algorithm 1 Figure 5shows a schematic diagram of the proposed phase-basedmotion detection algorithm
The algorithm is subdivided into two parts The first partcomputes local phase changes and the second part is thephase-based motion detector
In the first part the screen is divided into overlappingblocks For example the red green and blue blocks in theplane ldquodivide into overlapping blocksrdquo correspond to thesquares of the same color covering the video stream AGaussian window is then applied on each block followed bya 2D FFT operation that is used to extract the local phase Atemporal high-pass filter is then employed to extract phasechanges
In the second part the PMI is evaluated for each blockbased on the Radon transform of the local phase changes ineach block Motion is detected for blocks with PMI largerthan a preset threshold and the direction of motion iscomputed as in (39)
It is easy to notice that the algorithm can be highlyparallelized
333 Example We provide an illustrative example inFigure 6 showing how motion is detected using Algorithm 1The full video of this example can be found in SupplementaryVideo S1 see Supplementary Material available online athttpdxdoiorg10115520167915245 Figure 6(a) depicts astill from the ldquohighway videordquo in the Change Detection 2014dataset [22] evaluated at a particular time 119905
0 As suggested
by the algorithm the screen in Figure 6(a) is divided into26 times 19 overlapping blocks and the window functions areapplied to each block Local phases can then be extractedfrom the 2DFFT of eachwindowed block and the local phasechanges are obtained by temporal high-pass filtering Thephase change is shown in Figure 6(b) for all blocks with block(12 11) enlarged in Figure 6(c) and block (23 6) enlarged inFigure 6(d) (see also the plane ldquo2D FFT and extract phasechangerdquo in Figure 5) Note that at the time of the videoframe block (12 11) covers a part of the vehicle in motion
Computational Intelligence and Neuroscience 9
Input Visual stream or Video data 119906(119909 119910 119905)Output Detected MotionConstruct Gaussian window of size119872times119872 denote as 119908for 119905 = Δ119879 2Δ119879 119873Δ119879 do
Divide the screen of frame at time 119905 into multiple119872times119872 blocks 119906119896119897(119905) with overlaps
foreach block 119906119896119897(119905) do
Multiply pixel-wise the block 119906119896119897(119905) with the Gaussian window 119906
119896119897(119905)119908
Take119872times119872 2D FFT of 119906119896119897(119905)119908 denote phase as 120601
119896119897(120596119909 120596119910 119905) and amplitude as 119860
119896119897(120596119909 120596119910 119905)
foreach frequency (120596119909 120596119910) do
Highpass filter temporally 120601119896119897(120596119909 120596119910 119905) denote 1206011015840
119896119897(120596119909 120596119910 119905)
Optionally denoise by 1206011015840119896119897(120596119909 120596119910 119905) larr 120601
1015840
119896119897(120596119909 120596119910 119905)(119860
119896119897(120596119909 120596119910 119905)((1119872
2) sum(120596119909 120596119910)isinD
2 119860119896119897(120596119909 120596119910 119905) + 120598))
endCompute the radon transform (R120601
1015840
119896119897)(120588 120579 119905)
Compute PMI119896119897(119905) according to (38)
if PMI119896119897(119905) gt threshold then
Motion Detected at block (119896 119897) at time 119905Compute direction of motion according to (39)
endend
end
Algorithm 1 Phase-based motion detection algorithm using the FFT
y
120596x
120596y
y
t
120596x
120596y
x
y
x
y
x
x
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
Divided intooverlapping blocks
Apply window function
2D FFT andextract phase change
Radon transformand extract strength
and orientation of plane
Thresholding
Computephase
change
The phase-baseddetector
Figure 5 Schematic diagram of proposed phase-based motion detection algorithm
10 Computational Intelligence and Neuroscience
in the front and block (23 6) corresponds to an area of thehighway pavement where no motion occurs
Figure 6(f) depicts for each block (119896 119897) the maximumphase change over all (120596
119909 120596119910) isin D2 that is
max(120596119909120596119910)isinD2
10038161003816100381610038161003816100381610038161003816
119889120601119896119897
119889119905
(120596119909 120596119910 1199050)
10038161003816100381610038161003816100381610038161003816
(41)
We observe from the figure that for regions with lowamplitude such as the region depicting the road whenthe normalization constant is absent the derivative of thephase can be noisy For these blocks the maximum of|(119889120601119896119897119889119905)(120596
119909 120596119910 1199050)| over all (120596
119909 120596119910) isin D2 is comparable
to the maximum obtained for blocks that cover the vehiclesin motion
However (29) suggests that the local phase change frommultiple filter pairs centered at the same spatial position(119896 119897) can provide a constraint to robustly estimate motionand its direction Given the block structure employed in thecomputation of the local phase it is natural to utilize phasechange information from multiple sources
Indeed if for a particular block (119896 119897) (119889120601119896119897119889119905)(120596
119909
120596119910 119905) = minusV
119909(119905)120596119909minus V119910(119905)120596119910 then it is easy to see that (119889120601
119896119897
119889119905)(120596119909 120596119910 119905) will be zero on the line V
119909(119905)120596119909+ V119910(119905)120596119910= 0
and have opposite sign on either side of this line For examplein Figures 6(b) and 6(c) (119889120601
119896119897119889119905)(120596
119909 120596119910 1199050) clearly exhibits
this property for blocks that cover a vehicle in motion ThePMI is a tool to evaluate this property
Finally the PMIs for all blocks are shown compactlyin a heat map in Figure 6(e) The figure shows clearly thatthe blocks corresponding to the two moving vehicles have ahigh PMI value while the stationary background areas havea low PMI value allowing one to easily detect motion byemploying simple thresholding (see also the plane ldquoRadontransform and extract strength orientation of planerdquo inFigure 5) In addition the orientation ofmotion in each blockis readily observable even by inspection in Figure 6(b) by aline separating the yellow part and blue part in each blockFurther results about the direction of motion are presentedin Section 4
34 Relationship to Biological Motion Detectors A straight-forward way to implement local motion detectors is to applya complex-valued Gabor receptive field (5) to the video signal119906 and then take the derivative of the phase with respect totime or apply a high-pass filter on the phase to approximatethe derivative
We present here an alternate implementation withoutexplicitly computing the phaseThiswill elucidate the relationbetween the phase-based motion detector presented in theprevious section and some elementary motion detectionmodels used in biology such as theReichardtmotion detector[7] and motion energy detector [9]
Assuming that the local phase 12060100(120596119909 120596119910 119905) is differen-
tiable we have (see also the Appendix)
11988912060100(120596119909 120596119910 119905)
119889119905
=
(119889119887 (120596119909 120596119910 119905) 119889119905) 119886 (120596
119909 120596119910 119905) minus (119889119886 (120596
119909 120596119910 119905) 119889119905) 119887 (120596
119909 120596119910 119905)
[119886 (120596119909 120596119910 119905)]
2
+ [119887 (120596119909 120596119910 119905)]
2 (42)
where 119886(120596119909 120596119910 119905) and 119887(120596
119909 120596119910 119905) are respectively the real
and imaginary parts of 11986000(120596119909 120596119910 119905)11989011989512060100(120596119909120596119910119905)
We notice that the denominator of (42) is the square ofthe local amplitude of 119906 and the numerator is of the formof a second order Volterra kernel This suggests that thetime derivative of the local phase can be viewed as a secondorder Volterra kernel that processes two normalized spatiallyfiltered inputs V
1and V2
We consider an elaborated Reichardt motion detector asshown in Figure 7 It is equipped with a quadrature pair ofGabor filters whose outputs are 119903
1(119905) = 119886(120596
119909 120596119910 119905) and
1199032(119905) = 119887(120596
119909 120596119910 119905) respectively for a particular value of
(120596119909 120596119910) The pair of Gabor filters that provide these outputs
are the real and imaginary parts of119908(119909 119910)119890minus119895(120596119909119909+120596119910119910) It also
consists of a temporal high-pass 1198921(119905) and temporal low-
pass filter 1198922(119905) [9] The output of the elaborated Reichardt
detector follows the diagram in Figure 7 and can be expressedas
(1199032lowast 1198921) (119905) (119903
1lowast 1198922) (119905) minus (119903
1lowast 1198921) (119905) (119903
2lowast 1198922) (119905) (43)
The response can also be characterized by a second orderVolterra kernel We notice the striking similarity between
(43) and the numerator of (42) In fact the phase-basedmotion detector shares some properties with the Reichardtmotion detector For example it is straightforward to see thata single phase-basedmotion detector is tuned to the temporalfrequency of a moving sinusoidal grating
Since the motion energy detector is formally equivalentto an elaborated Reichardt motion detector [9] the structureof the motion energy detector with divisive normalization isalso similar to the phase-based motion detector
4 Exploratory Results
In this section we apply the phase-based motion detectionalgorithm on several video sequences and demonstrate itsefficiency and effectiveness in detecting local motion Themotion detection algorithm is compared to two well-knownbiological motion detectors namely the Reichardt motiondetector and the Barlow-Levick motion detector We thenshow that the detected local motion can be used in motionsegmentation tasks The effectiveness of the segmentationis compared to segmentation using motion informationobtained from a widely used optic flow based algorithmavailable in the literature [4]
Computational Intelligence and Neuroscience 11
(f) (e)
(b)
(a)
(d)
(c)
minus120587 minus1205872 0 1205872
10080604020
y (pi
xel)
100 15050x (pixel)
0020406081
15
10
5
l
01234567+
15
10
5
l
0510152025+
minus6
minus4
minus2
0
2
4
6
181716151413121110
9876543210
l
2 4 6 8 10 12 14 16 18 20 22 240k
minus6minus4minus20246
minus6minus4minus20246
120596x
minus120587 minus1205872 0 1205872
120596x
1205872
0
minus1205872
minus120587
120596y
1205872
0
minus1205872
minus120587
120596y
10 15 20 255k
10 20 255 15k
Figure 6 Evaluating the motion detection algorithm at time 1199050(see also Supplementary Video S1) (a) A still from a 156 times 112-pixel video at
time 1199050 (b) (119889120601
119896119897119889119905)(120596
119909 120596119910 1199050) for all 26times19 blocks each of size 32times32The blocks are concatenated to create a large ldquophase imagerdquo with their
relative neighbors kept (c) (1198891206011211
119889119905)(120596119909 120596119910 1199050) shown in the (120596
119909 120596119910) isin D2 space (d) (119889120601
236119889119905)(120596
119909 120596119910 1199050) shown in the (120596
119909 120596119910) isin D2
space (e) Phase Motion Indicator (PMI) for all blocks at 1199050 Each pixel (119896 119897) is color coded for PMI
119896119897 (f) max
(120596119909 120596119910)isinD2 |(119889120601119896119897119889119905)(120596
119909 120596119910 1199050)|
for all blocks at 1199050 Each pixel (119896 119897) is color coded for this maximum value of block (119896 119897)
g1(t)g2(t) g1(t) g2(t)
times times
minus++
SpatialGaborfilter
SpatialGaborfilter
r1(t) r2(t)
u(x y t)
Figure 7 Diagram of an elaborated Reichardt motion detector Itconsists of a quadrature pair of Gabor filters whose outputs 119903
1and 1199032
provide inputs to the high-pass filters 1198921(119905) and the low-pass filters
1198922(119905)
Table 1 Processing Speeds of the proposed algorithm the Reichardtmotion detector and the Barlow-Levick Motion detector
Screen size Processing capability (frames per second)Proposed Reichardt Barlow-Levick
320 times 240 420 790 800720 times 576 135 685 6901280 times 720 60 274 2931920 times 1080 27 191 194
41 Efficient Parallel Implementation The algorithm wasimplemented in PyCUDA [23] and tested on an NVIDIAGeForce GTX TITAN GPU All computations use singleprecision floating points The processing speeds of the algo-rithm for several screen sizes are listed in Table 1 Clearlythe proposed phase-based motion detection algorithm hasreal-time capability to process video even with full HighDefinition screen size
For comparison we implemented the Reichardt motiondetector and the Barlow-Levick motion detector Theirrespective diagrams are shown in Figure 8 Note that we
12 Computational Intelligence and Neuroscience
ukl(t) uk+1l(t)
g1(t)
g2(t)
g1(t)
g2(t)
+ minus+
timestimes
(a)
minus
minus minus
+
ukl(t) uk+1l(t)
g1(t)g1(t)
g2(t) g2(t)
++
+
(b)
Figure 8 (a) A Reichardt motion detector detecting motion between the two points 119906119896119897and 119906
119896+1119897in the blurred and downsampled video
(b) A Barlow-Levick motion detector detecting motion between the two points 119906119896119897and 119906
119896+1119897in the blurred and downsampled video 119892
1(119905)
denotes a high-pass filter and 1198922(119905) denotes a low-pass filter
moved the high-pass filter ℎ1(119905) to the front of the low-pass fil-
ter ℎ2(119905) This configuration provides a superior performance
to the one in Figure 7 For both the Reichardt and the Barlow-Levick detectors the videos are first blurred by a Gaussianfilter (with the same variance as the Gaussian window in (30)used for the phase-based motion detector) and subsampledat the center of each overlapping block in the phase-basedmotion detector The subsampled video then provides inputsto two 2D arrays of the circuits shown in Figure 8 one forthe horizontal direction and one for the vertical directionThe outputs of the horizontal and vertical motion circuitsform an array of motion vectors that indicate the strengthand direction of motion The three tested motion detectorshave the same number of outputs as a result The processingspeeds of the Reichardt motion detector and the Barlow-Levick motion detector both implemented in PyCUDA andtested on the same GPU are shown in Table 1
Note that the Reichardt motion detector and the Barlow-Levick motion detector are highly efficient due to the sim-plicity of their algorithmsThe phase-basedmotion detectionalgorithm however is a much more sophisticated algorithmand yet it can be implemented in real-time using parallelcomputing devices
The fast GPU implementation is based on the FFTand Matrix-Matrix multiplication It is expected that thoseoperations can be efficiently implemented in hardware forexample FPGA
42 Examples of Phased-BasedMotionDetection Weappliedour motion detection algorithm on video sequences of theChange Detection 2014 dataset [22] that did not exhibitcamera egomotion For these video sequences the standarddeviation of the Gaussian window functions was set to 4
pixels and the block size was chosen to be 32 times 32 pixelsThreshold and normalization parameters were kept the samewith the exception of the ldquothermal videordquo (in order to dealwith larger background noise levels see below) We alsotested the same video sequences using the Reichardt motiondetector and the Barlow-Levick motion detector For theReichardt detector the high-pass filters were chosen as firstorder filters with a time constant of 200 milliseconds and thelow-pass filters were chosen as first order filters with a timeconstant of 300 milliseconds (assuming that the frame rate is50 frames per second)Thresholdwas set to 2 For the Barlow-Levick motion detector the time constant of the first high-pass filters was set to 250 milliseconds The low-pass filterswere the same as in the Reichardt detectorThreshold was setto 2
The first video was taken from a highway surveillancecamera (ldquohighway videordquo) under good illumination con-ditions and high contrast The video had moderate noiseparticularly on the road surface The detected motion isshown in the top left panel of Figure 9 (see SupplementaryVideo S2 for full video) The phase-based motion detectionalgorithm captured both the moving cars and the tree leavesmoving (due to the wind) In the time interval between the9th and 10th second the camera was slightly moved left-and right-wards within 5 frames again possibly due to thewind Movement due to this shift was captured by themotiondetection algorithm and the algorithm was fast enough tocorrectly determine the direction of this movement In thisvideo we already noted that this algorithm suffers from theaperture problem For example in front of the van where along horizontal edge is present the detectedmotion ismostlypointing downwards In addition to moving downwards theedge is alsomoving to the left howeverThis is expected since
Computational Intelligence and Neuroscience 13
(a) (b) (c)
Figure 9 Top motion detection of the ldquohighway videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) Red arrows indicate detected motion and its direction Bottom the contrast of the video was artificiallyreduced by 5-fold and the mean was reduced to 35 of the original The red arrows are duplicated from the motion detection result on theoriginal video as in top Blue arrows are the result of motion detection on the video with reduced contrast If motion is detected both in theoriginal video and in the video with reduced contrast then the arrow is shown in magenta (as a mix of blue and red) (see SupplementaryVideo S2 for full video)
the algorithm only detects motion locally and does not takeinto account the overall shape of any object
For comparison the motion detection results for theReichardt motion detector and the Barlow-Levick motiondetector are shown in the top middle and top right panelof Figure 9 (see Supplementary Video S2 for full video) TheReichardt detector performed relatively well when vehiclesare moving faster but the direction it predicts for vehiclesmoving slower for example on the back of the image is notaccurate In addition motion is still detected in some partsof the screen where the vehicles have just passed by Thedetection result for the Barlow-Levick motion detector waspoorer In particular the response to OFF edge movement isalways the opposite to the actual movement direction
We then squeezed the range of the screen intensity from[0 1] to [02 04] resulting in a video with a lower meanluminance and lower contrast The motion detection resultson the low-contrast video are shown in the bottom 3 panelsof Figure 9 (see Supplementary Video S2 for full video) Forreference the motion detection results for the original videoare shown in red arrows Motion detected in the low-contrastvideo is shown in blue arrows if no motion is detected forthe block in the original video If motion is detected in boththe original and the low-contrast video the arrows are shownin magenta Figure 9 clearly shows that while the motiondetection performance is degraded for all three detectors thephase-basedmotion detector performed still quite well in thelow-contrast video detecting most of the moving vehicles
The other two detectors missed many of the blocks wheremotion was detected in the original movie
To quantify how well each detector works under differentcontrast conditions we computed the ratio of unthresholdedoutput values of eachmotion detector between lower contrastand full contrast video For example in the case of phase-basedmotion detector we computed for each block the ratiobetween the PMI index for the lower contrast video and thatfor the full contrast videoThe ratios are then averaged acrossall blocks where motion is detected in the full contrast videoThis average for different contrasts is shown in Figure 10 bythe blue curve In the ideal case when the normalizationconstant 120598 is 0 the phase detector should produce invariantresponses to videos with different contrast The curve shownhere is mainly due to a nonzero 120598The ratios for the Reichardtmotion detector and the Barlow-Levick motion detectorare computed similarly and are shown in red and yellowrespectively It is clear that the phase-based motion detectorhas a superior performance across a range of contrast valuesAt 20 contrast the phase-based detector still has 50 ofthe PMI index value for full contrast video As expectedthe response of Barlow-Levick motion detector is linear withrespect to contrast and the Reichardt motion detector has aquadratic relation to contrast For a fixed threshold value alarger ratio equates to amore consistent performance at lowercontrast
The second video was captured by a surveillance camerain a train station (ldquotrain station videordquo) The video was
14 Computational Intelligence and Neuroscience
Phase-basedReichardtBarlow-Levick
0
02
04
06
08
1
Ratio
40 60 80200 100Contrast ()
Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined
under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm
The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)
The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression
With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)
43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient
objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area
For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold
We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually
We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm
We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)
These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm
5 Discussion
Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes
Computational Intelligence and Neuroscience 15
(a) (b) (c)
Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)
(a) (b) (c)
Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)
by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to
models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]
We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in
16 Computational Intelligence and Neuroscience
(a) (b) (c)
Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)
Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)
phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in
Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)
modeling in vivo biological vision systems This is consistentwith other recent findings [15]
Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure
Computational Intelligence and Neuroscience 17
(a) (b)
Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)
(a) (b)
Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)
(a) (b)
Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)
Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase
itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third
18 Computational Intelligence and Neuroscience
(a) (b)
Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)
instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection
The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions
Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors
We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well
We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It
can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes
Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems
Appendix
The Derivative of the Local Phase
Let
119886 (120596119909 120596119910 119905)
= int
R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596
119909119909 + 120596119910119910) 119889119909 119889119910
119887 (120596119909 120596119910 119905)
= minusint
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
(A1)
and therefore
11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)
= 119886 (120596119909 120596119910 119905) + 119895119887 (120596
119909 120596119910 119905)
(A2)
The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596
119909
120596119910 119905) 119886(120596
119909 120596119910 119905)) and therefore
11988912060100
119889119905
=
119886 (119889119887119889119905) minus 119887 (119889119886119889119905)
[119886]2+ [119887]2
(A3)
Computational Intelligence and Neuroscience 19
Now
119889119886
119889119905
(120596119909 120596119910 119905)
= int
R2
120597119906
120597119905
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
(A4)
For a purely translational signal with instantaneous spatialshift given by (119904
119909(119905) 119904119910(119905)) within the support of the window
we have
119889119886
119889119905
(120596119909 120596119910 119905)
= minusint
R2
119889119904119909
119889119905
(119905)
120597119906
120597119909
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus int
R2
119889119904119910
119889119905
(119905)
120597119906
120597119910
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119909
119889119904119909
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119910
119889119904119910
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199033=1199033(120596119909120596119910119905)
+ 120596119909
119889119904119909
119889119905
119887
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199034=1199034(120596119909120596119910119905)
+ 120596119910
119889119904119910
119889119905
119887
(A5)
Similarly
119889119887
119889119905
(120596119909 120596119910 119905) = minus
119889119904119909
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199035=1199035(120596119909120596119910119905)
minus 120596119909
119889119904119909
119889119905
119886 minus
119889119904119910
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199036=1199036(120596119909120596119910119905)
minus 120596119910
119889119904119910
119889119905
119886
(A6)
Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to
11988912060100
119889119905
(120596119909 120596119910 119905) =
minus (119889119904119909119889119905) [119903
3119887 + 1199035119886 + 120596119909 [
119886]2+ 120596119909 [
119887]2] minus (119889119904
119910119889119905) [119903
4119887 + 1199036119886 + 120596119910 [
119886]2+ 120596119910 [
119887]2]
[119886]2+ [119887]2
= minus
119889119904119909
119889119905
120596119909minus
119889119904119910
119889119905
120596119910minus
119889119904119909
119889119905
sdot
1199033119887 + 1199035119886
[119886]2+ [119887]2minus
119889119904119910
119889119905
sdot
1199034119887 + 1199036119886
[119886]2+ [119887]2
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
v00(120596119909120596119910119905)
(A7)
Disclosure
The authorrsquos names are alphabetically listed
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgment
The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232
References
[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010
[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010
[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014
[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010
20 Computational Intelligence and Neuroscience
[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013
[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014
[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956
[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010
[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985
[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965
[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981
[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015
[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014
[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999
[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015
[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001
[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996
[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980
[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013
[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013
[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014
[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014
[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach
to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012
[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990
[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987
[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998
Submit your manuscripts athttpwwwhindawicom
Computer Games Technology
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Distributed Sensor Networks
International Journal of
Advances in
FuzzySystems
Hindawi Publishing Corporationhttpwwwhindawicom
Volume 2014
International Journal of
ReconfigurableComputing
Hindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Applied Computational Intelligence and Soft Computing
thinspAdvancesthinspinthinsp
Artificial Intelligence
HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014
Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Journal of
Computer Networks and Communications
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation
httpwwwhindawicom Volume 2014
Advances in
Multimedia
International Journal of
Biomedical Imaging
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
ArtificialNeural Systems
Advances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience
Industrial EngineeringJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Human-ComputerInteraction
Advances in
Computer EngineeringAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience 5
(a) (b) (c)
Figure 2 An example of reconstruction of image from local phase information (a) Original image the dimension of the space is 16129(b) Reconstructed image from 20102 phase measurements scaled to match the original (c) Error
3 Visual Motion Detection fromPhase Information
In this section we consider visual fields that change as afunction of time For notational simplicity 119906 will denote herethe space-time intensity of the visual field
31 The Global Phase Equation for Translational Motion Let119906 = 119906(119909 119910 119905) (119909 119910) isin R2 119905 isin R be a visual stimulus If thevisual stimulus is a pure translation of the signal at 119906(119909 119910 0)that is
119906 (119909 119910 119905) = 119906 (119909 minus 119904119909(119905) 119910 minus 119904
119910(119905) 0) (22)
where
119904119909(119905) = int
119905
0
V119909(119904) 119889119904
119904119910(119905) = int
119905
0
V119910(119904) 119889119904
(23)
are the total length of translation at time 119905 in each dimen-sion and V
119909(119905) and V
119910(119905) are the corresponding instanta-
neous velocity components then the only difference between119906(119909 119910 119905) and 119906(119909 119910 0) in the Fourier domain is captured bytheir global phase More formally consider the following
Lemma 6 The change (derivative) of the global phase is givenby
119889120601 (120596119909 120596119910 119905)
119889119905
= minus120596119909V119909(119905) minus 120596
119910V119910(119905)
= minus [120596119909 120596119910] [V119909(119905) V119910(119905)]
119879
(24)
where by abuse of notation 120601(120596119909 120596119910 119905) denotes the global
phase of 119906(119909 119910 119905) and 120601(120596119909 120596119910 0) is the initial condition
Proof If (F119906(sdot sdot 0)) (120596119909 120596119910) is the 2D (spatial) Fourier
transform of 119906 = 119906(119909 119910 0) (119909 119910) isin R2 by the Fourier shifttheorem we have
(F119906 (sdot sdot 119905)) (120596119909 120596119910)
= (F119906 (sdot sdot 0)) (120596119909 120596119910) 119890minus119895(120596119909119904119909(119905)+120596
119910119904119910(119905))
(25)
For a certain frequency component (120596119909 120596119910) the change of its
global phase over time amounts to
119889120601 (120596119909 120596119910 119905)
119889119905
= minus120596119909
119889119904119909(119905)
119889119905
minus 120596119910
119889119904119910(119905)
119889119905
= minus120596119909V119909(119905) minus 120596
119910V119910(119905)
= minus [120596119909 120596119910] [V119909(119905) V119910(119905)]
119879
(26)
Therefore in the simple case where the entire visual field isshifting the derivative of the phase of Fourier componentsindicates motion and it can be obtained by the innerproduct between the component frequency and the velocityvector
32 The Change of Local Phase
321The Local Phase Equation for Translational Motion Theanalysis in Section 31 applies to global motion This type ofmotion occurs most frequently when the imaging deviceeither an eye or a cameramoves Visualmotion in the naturalenvironment however ismore diverse across the screen sinceit is often time produced by multiple moving objects Theobjects can be small and the motion more localized in thevisual field
Taking the global phase of 119906 will not simply revealwhere motion of independent objects takes place or theirdirectionvelocity of motion The ease of interpretation ofmotion by using the Fourier transform however motivates
6 Computational Intelligence and Neuroscience
us to reuse the same concept in detecting local motion Thiscan be achieved by restricting the domain of the visual fieldwhere the Fourier transform is applied
To be able to detect local motion we consider thelocal phase of 119906(119909 119910 119905) by taking the STFT with windowfunction 119908(119909 119910) Note that the STFT and its ubiquitousimplementation in DSP chips can be extensively used in anydimension For simplicity and without loss of generality weconsider the window to be centered at (0 0) The STFT isgiven by
int
R2119906 (119909 119910 119905) 119908 (119909 119910) 119890
minus119895(120596119909119909+120596119910119910)119889119909 119889119910
= 11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)
(27)
where by abuse of notation 11986000(120596119909 120596119910 119905) is the amplitude
and 12060100(120596119909 120596119910 119905) the local phase
Before we move on to the mathematical analysis we canintuitively explain the relation between the change in localphase and visual motion taking place across the windowsupport First if the stimulus undergoes a uniform changeof intensity or it changes proportionally over time due tolighting conditions for example the local phase does notchange since the phase is invariant with respect to intensityscaling Therefore the local phase does not change for suchnonmotion stimuli Second a rigid edge moving across thewindow support will induce a phase change
For a strictly translational signal within the windowsupport (footprint) for example
119906 (119909 119910 119905) = 119906 (119909 minus 119904119909(119905) 119910 minus 119904
119910(119905) 0)
for (119909 119910) isin supp (119908 (119909 119910))
(28)
where 119904119909(119905) and 119904
119910(119905) are as defined in (23) we have the
following result
Lemma 7 Consider the following
11988912060100
119889119905
(120596119909 120596119910 119905) = minus
119889119904119909(119905)
119889119905
120596119909minus
119889119904119910(119905)
119889119905
120596119910
+ v00(120596119909 120596119910 119905)
(29)
where by abuse of notation 12060100(120596119909 120596119910 119905) is the local phase
of 119906(119909 119910 119905) and 12060100(120596119909 120596119910 0) is the initial condition The
functional form of the term v00
= v00(120596119909 120596119910 119905) is provided
in the Appendix
Proof The derivation of (29) and the functional form of v00
are given in the Appendix
Remark 8 Wenotice that the derivative of the local phase hassimilar structure to that of the global phase but for the addedterm v
00Through simulations we observed that the first two
terms in (29) dominate over the last term for an ON or OFFmoving edge [3] For example Figure 3 shows the derivativeof the local phase given in (29) for an ON edge moving withvelocity (40 0) pixelssec
42
0minus2
minus4
100
d120601dt
120596yminus250minus200minus150minus100minus50
050
150
200250
31 2minus4 0minus1minus3 minus2 4
120596x
Figure 3 Derivative of the local phase for an ON edge moving inthe positive 119909 direction
Remark 9 Note that 12060100(120596119909 120596119910 119905) may not be differen-
tiable even if 119906(119909 119910 119905) is differentiable particularly when11986000(120596119909 120596119910 119905) = 0 For example the spatial phase can jump
from a positive value to zero when11986000(120596119909 120596119910 119905) diminishes
This also suggests that the instantaneous local spatial phaseis less informative about a region of a visual scene whenever11986000(120596119909 120596119910 119905) is close to zero Nevertheless the time deriva-
tive of the local phase can be approximated by applying ahigh-pass filter to 120601
00(120596119909 120596119910 119905)
322 The Block Structure for Computing the Local Phase Weconstruct Gaussian windows along the 119909 119910 dimensions TheGaussian windows are defined as
(T119896119897119908) (119909 119910) = 119890
minus((119909minus119909119896)2+(119909minus119910
119897)2)21205902
(30)
where 119909119896= 1198961198870 119910119897= 1198971198870 in which 119887
0isin Z+ is the distance
between two neighboring windows and 1 le 1198961198870le 119875119909 1 le
1198971198870le 119875119910 where 119875
119909 119875119910isin Z+ are the number of pixels of the
screen in 119909 and 119910 directions respectivelyWe then take the 2D Fourier transform of the windowed
video signal 119906(119909 119910 119905)(T119896119897119908)(119909 119910) and write in polar form
int
R2119906 (119909 119910 119905) (T
119896119897119908) (119909 119910) 119890
minus119895(120596119909(119909minus119909119896)+120596119910(119910minus119910119897))119889119909 119889119910
= 119860119896119897(120596119909 120596119910 119905) 119890119895120601119896119897(120596119909120596119910119905)
(31)
The above integral can be very efficiently evaluated usingthe 2D FFT in discrete domain defined on 119872 times 119872 blocksapproximating the footprint of the Gaussian windows Forexample the standard deviation of the Gaussian windows weuse in the examples in Section 4 is 4 pixels A block of 32times32
pixels (119872 = 32) is sufficient to cover the effective support(or footprint) of the Gaussian window At the same time thesize of the block is a power of 2 which is most suitable forFFT-based computationThe processing of each block (119896 119897) isindependent of all other blocks thereby parallelism is readilyachieved
Note that the size of the window is informed by thesize of the objects one is interested in locating Measure-ments of the local phase using smaller window functions
Computational Intelligence and Neuroscience 7
60
50
40
30
20
10
y (pi
xel)
01
02
03
04
05
06
07
08
09
1
20 30 40 50 6010x (pixel)
(a)
0
02
04
06
08
1
Mag
nitu
de (a
u)
10 20 30 40 50 60 700y (pixel)
(b)
Figure 4 Example of block structure (a) Four yellow disks show 4 Gaussian window functions with translation parameter 119896 = 3 119897 = 3
(top-left) 119896 = 3 119897 = 7 (bottom-left) 119896 = 7 119897 = 3 (top-right) and 119896 = 7 119897 = 7 (bottom-right) The red solid square shows a 32 times 32-pixel blockapproximating the Gaussian window with 119896 = 3 119897 = 3 and the green dashed square shows a 32 times 32-pixel block approximating the Gaussianwindow with 119896 = 3 119897 = 7 (b) Cross section of all 11 Gaussian windows with translation parameters 119896 = 7 119897 isin [0 10] The cross section istaken as indicated by the magenta line in (a) and the red and blue curve correspond to cross sections of the two Gaussian windows shown in(a) centered on the magenta line
are less robust to noise Larger windows would enhanceobject motion detection if the object size is comparable tothe window size However there would be an increasedlikelihood of independent movement of multiple objectswithin the same window which is not modeled here andthereby may not be robustly detected
Therefore for each block (119896 119897) we obtain 1198722 measure-
ments of the phase 120601119896119897(120596119909119898
120596119910119898
119905) at every time instant 119905with (120596
119909119898
120596119910119898
) isin D2 where
D2= (120596
119909119898
= 1198981205960 120596119910119899
= 1198991205960) 119898 119899 = minus1198722
minus 1198722 + 1 1198722 minus 1
(32)
with 1205960= 2120587119872 We then compute the temporal derivative
of the phase that is (119889120601119896119897119889119905)(120596
119909 120596119910 119905) for (120596
119909 120596119910) isin D2
We further illustrate an example of the block structurein Figure 4 Figure 4(a) shows an example of an image of64 times 64 pixels Four Gaussian windows are shown each witha standard deviation of 4 pixels The distance between thecenters of two neighboring Gaussian windows is 6 pixelsThe red solid square shows a 32 times 32-pixel block with 119896 =
3 119897 = 3 which encloses effective support of the Gaussianwindow on top-left (119896 = 0 119897 = 0 is the block with Gaussianwindow centered at pixel (1 1)) The green dashed squareshows another 32 times 32-pixel block with 119896 = 3 119897 = 7 Thetwo Gaussian windows on the right are associated with theblocks 119896 = 7 119897 = 3 and 119896 = 7 119897 = 7 respectively Crosssection of all Gaussian windows with 119896 = 7 119897 isin [0 10] that isthose centered on themagenta line are shown in Figure 4(b)The red and blue curve in Figure 4(b) correspond to the two
Gaussian windows shown in Figure 4(a) Figure 4(b) alsosuggests that some of the Gaussian windows are cut off on theboundaries This is however equivalent to assuming that thepixel values outside the boundary are always zero and it willnot significantly affect motion detection based on the changeof local phase
Since the phase and thereby the phase change is noisierwhen the local amplitude is low an additional denoisingstep can be employed to discount the measurements of(119889120601119896119897119889119905)(120596
119909 120596119910 119905) for low amplitude values 119860
119896119897(120596119909 120596119910 119905)
The denoising is given by
119889120601119896119897
119889119905
(120596119909 120596119910 119905)
sdot
119860119896119897(120596119909 120596119910 119905)
(11198722)sum(120596119909120596119910)isinD2 119860119896119897 (120596119909 120596119910 119905) + 120598
(33)
where 120598 gt 0 is a constant and (120596119909 120596119910) isin D2
33 The Phase-Based Detector We propose here a block FFTbased algorithm to detect motion using phase informationSuch an algorithm is due to its simplicity and parallelismhighly suitable for an in silico implementation
331 Radon Transform on the Change of Phases We exploitthe approximately linear structure of the phase derivative forblocks exhibiting motion by computing the Radon transformof (119889120601
119896119897119889119905)(120596
119909 120596119910 119905) over a circular bounded domain 119862 =
(120596119909 120596119910) | (120596119909 120596119910) isin D2 1205962
119909+ 1205962
119910lt 1205872
8 Computational Intelligence and Neuroscience
The Radon transform of the change of phase in thedomain 119862 is given by
(R119889120601119896119897
119889119905
) (120588 120579 119905) = int
R
119889120601119896119897
119889119905
(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588
sdot sin 120579 + 119904 sdot cos 120579 119905) 1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579
+ 119904 sdot cos 120579) 119889119904
(34)
where
1119862(120596119909 120596119910) =
1 if (120596119909 120596119910) isin 119862
0 otherwise(35)
The Radon transform (R(119889120601119896119897119889119905))(120588 120579 119905) evaluated at
a particular point (1205880 1205790 1199050) is essentially an integral
of (119889120601119896119897119889119905)(120596
119909 120596119910 1199050) along a line oriented at angle
1205872 + 1205790with the 120596
119909axis and at distance |120588
0| along the
(cos(1205790) sin(120579
0)) direction from (0 0)
If for a particular 119896 and 119897 we have (119889120601119896119897119889119905)(120596
119909 120596119910 119905) =
minusV119909(119905)120596119909minus V119910(119905)120596119910 we have
(R (119889120601119896119897119889119905)) (120588 120579 119905)
c (120588 120579)
= 120588 [minusV119909(119905) cos 120579 minus V
119910(119905) sin 120579]
(36)
where
c (120588 120579)
= int
R
1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579 + 119904 sdot cos 120579) 119889119904
(37)
c(120588 120579) is a correction term due to different length of lineintegrals for different values of (120588 120579) in the bounded domain119862
After computing the Radon transform of (119889120601119896119897119889119905)(120596
119909
120596119910 119905) for every block (119896 119897) at time 119905
0 we compute the Phase
Motion Indicator (PMI) defined as
PMI119896119897= max120579isin[0120587)
sum
120588
100381610038161003816100381610038161003816100381610038161003816
(R (119889120601119896119897119889119905)) (120588 120579 119905
0)
c (120588 120579)
100381610038161003816100381610038161003816100381610038161003816
(38)
If the PMI119896119897
is larger than a chosen threshold motion isdeemed to occur in block (119896 119897) at time 119905
0
Using the Radon transform makes it easier to separaterigid motion from noise Since the phase is quite sensitiveto noise particularly when the amplitude is very smallthe change of phase under noise may have comparablemagnitude to that due to motion as mentioned earlier Thechange of phase under noise however does not possess thestructure suggested by (29) in the (120596
119909 120596119910) domain Instead
it appears to be more randomly distributed Consequently
the PMI value is comparatively small for these blocks (see alsoSection 333)
Moreover the direction of motion for block (119896 119897) wheremotion is detected can be easily computed as
120579119896119897
= 120587(
sign (sum120588gt0
(R (119889120601119896119897119889119905)) (120588 120572
119896119897 1199050) c (120588 120572
119896119897)) + 1
2
)
+ 120572119896119897
(39)
where
120572119896119897= argmax120579isin[0120587)
sum
120588
100381610038161003816100381610038161003816100381610038161003816
(R (119889120601119896119897119889119905)) (120588 120579 119905
0)
c (120588 120579)
100381610038161003816100381610038161003816100381610038161003816
(40)
This follows from (36)
332 The Phase-Based Motion Detection Algorithm We for-mally summarize the above analysis as Algorithm 1 Figure 5shows a schematic diagram of the proposed phase-basedmotion detection algorithm
The algorithm is subdivided into two parts The first partcomputes local phase changes and the second part is thephase-based motion detector
In the first part the screen is divided into overlappingblocks For example the red green and blue blocks in theplane ldquodivide into overlapping blocksrdquo correspond to thesquares of the same color covering the video stream AGaussian window is then applied on each block followed bya 2D FFT operation that is used to extract the local phase Atemporal high-pass filter is then employed to extract phasechanges
In the second part the PMI is evaluated for each blockbased on the Radon transform of the local phase changes ineach block Motion is detected for blocks with PMI largerthan a preset threshold and the direction of motion iscomputed as in (39)
It is easy to notice that the algorithm can be highlyparallelized
333 Example We provide an illustrative example inFigure 6 showing how motion is detected using Algorithm 1The full video of this example can be found in SupplementaryVideo S1 see Supplementary Material available online athttpdxdoiorg10115520167915245 Figure 6(a) depicts astill from the ldquohighway videordquo in the Change Detection 2014dataset [22] evaluated at a particular time 119905
0 As suggested
by the algorithm the screen in Figure 6(a) is divided into26 times 19 overlapping blocks and the window functions areapplied to each block Local phases can then be extractedfrom the 2DFFT of eachwindowed block and the local phasechanges are obtained by temporal high-pass filtering Thephase change is shown in Figure 6(b) for all blocks with block(12 11) enlarged in Figure 6(c) and block (23 6) enlarged inFigure 6(d) (see also the plane ldquo2D FFT and extract phasechangerdquo in Figure 5) Note that at the time of the videoframe block (12 11) covers a part of the vehicle in motion
Computational Intelligence and Neuroscience 9
Input Visual stream or Video data 119906(119909 119910 119905)Output Detected MotionConstruct Gaussian window of size119872times119872 denote as 119908for 119905 = Δ119879 2Δ119879 119873Δ119879 do
Divide the screen of frame at time 119905 into multiple119872times119872 blocks 119906119896119897(119905) with overlaps
foreach block 119906119896119897(119905) do
Multiply pixel-wise the block 119906119896119897(119905) with the Gaussian window 119906
119896119897(119905)119908
Take119872times119872 2D FFT of 119906119896119897(119905)119908 denote phase as 120601
119896119897(120596119909 120596119910 119905) and amplitude as 119860
119896119897(120596119909 120596119910 119905)
foreach frequency (120596119909 120596119910) do
Highpass filter temporally 120601119896119897(120596119909 120596119910 119905) denote 1206011015840
119896119897(120596119909 120596119910 119905)
Optionally denoise by 1206011015840119896119897(120596119909 120596119910 119905) larr 120601
1015840
119896119897(120596119909 120596119910 119905)(119860
119896119897(120596119909 120596119910 119905)((1119872
2) sum(120596119909 120596119910)isinD
2 119860119896119897(120596119909 120596119910 119905) + 120598))
endCompute the radon transform (R120601
1015840
119896119897)(120588 120579 119905)
Compute PMI119896119897(119905) according to (38)
if PMI119896119897(119905) gt threshold then
Motion Detected at block (119896 119897) at time 119905Compute direction of motion according to (39)
endend
end
Algorithm 1 Phase-based motion detection algorithm using the FFT
y
120596x
120596y
y
t
120596x
120596y
x
y
x
y
x
x
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
Divided intooverlapping blocks
Apply window function
2D FFT andextract phase change
Radon transformand extract strength
and orientation of plane
Thresholding
Computephase
change
The phase-baseddetector
Figure 5 Schematic diagram of proposed phase-based motion detection algorithm
10 Computational Intelligence and Neuroscience
in the front and block (23 6) corresponds to an area of thehighway pavement where no motion occurs
Figure 6(f) depicts for each block (119896 119897) the maximumphase change over all (120596
119909 120596119910) isin D2 that is
max(120596119909120596119910)isinD2
10038161003816100381610038161003816100381610038161003816
119889120601119896119897
119889119905
(120596119909 120596119910 1199050)
10038161003816100381610038161003816100381610038161003816
(41)
We observe from the figure that for regions with lowamplitude such as the region depicting the road whenthe normalization constant is absent the derivative of thephase can be noisy For these blocks the maximum of|(119889120601119896119897119889119905)(120596
119909 120596119910 1199050)| over all (120596
119909 120596119910) isin D2 is comparable
to the maximum obtained for blocks that cover the vehiclesin motion
However (29) suggests that the local phase change frommultiple filter pairs centered at the same spatial position(119896 119897) can provide a constraint to robustly estimate motionand its direction Given the block structure employed in thecomputation of the local phase it is natural to utilize phasechange information from multiple sources
Indeed if for a particular block (119896 119897) (119889120601119896119897119889119905)(120596
119909
120596119910 119905) = minusV
119909(119905)120596119909minus V119910(119905)120596119910 then it is easy to see that (119889120601
119896119897
119889119905)(120596119909 120596119910 119905) will be zero on the line V
119909(119905)120596119909+ V119910(119905)120596119910= 0
and have opposite sign on either side of this line For examplein Figures 6(b) and 6(c) (119889120601
119896119897119889119905)(120596
119909 120596119910 1199050) clearly exhibits
this property for blocks that cover a vehicle in motion ThePMI is a tool to evaluate this property
Finally the PMIs for all blocks are shown compactlyin a heat map in Figure 6(e) The figure shows clearly thatthe blocks corresponding to the two moving vehicles have ahigh PMI value while the stationary background areas havea low PMI value allowing one to easily detect motion byemploying simple thresholding (see also the plane ldquoRadontransform and extract strength orientation of planerdquo inFigure 5) In addition the orientation ofmotion in each blockis readily observable even by inspection in Figure 6(b) by aline separating the yellow part and blue part in each blockFurther results about the direction of motion are presentedin Section 4
34 Relationship to Biological Motion Detectors A straight-forward way to implement local motion detectors is to applya complex-valued Gabor receptive field (5) to the video signal119906 and then take the derivative of the phase with respect totime or apply a high-pass filter on the phase to approximatethe derivative
We present here an alternate implementation withoutexplicitly computing the phaseThiswill elucidate the relationbetween the phase-based motion detector presented in theprevious section and some elementary motion detectionmodels used in biology such as theReichardtmotion detector[7] and motion energy detector [9]
Assuming that the local phase 12060100(120596119909 120596119910 119905) is differen-
tiable we have (see also the Appendix)
11988912060100(120596119909 120596119910 119905)
119889119905
=
(119889119887 (120596119909 120596119910 119905) 119889119905) 119886 (120596
119909 120596119910 119905) minus (119889119886 (120596
119909 120596119910 119905) 119889119905) 119887 (120596
119909 120596119910 119905)
[119886 (120596119909 120596119910 119905)]
2
+ [119887 (120596119909 120596119910 119905)]
2 (42)
where 119886(120596119909 120596119910 119905) and 119887(120596
119909 120596119910 119905) are respectively the real
and imaginary parts of 11986000(120596119909 120596119910 119905)11989011989512060100(120596119909120596119910119905)
We notice that the denominator of (42) is the square ofthe local amplitude of 119906 and the numerator is of the formof a second order Volterra kernel This suggests that thetime derivative of the local phase can be viewed as a secondorder Volterra kernel that processes two normalized spatiallyfiltered inputs V
1and V2
We consider an elaborated Reichardt motion detector asshown in Figure 7 It is equipped with a quadrature pair ofGabor filters whose outputs are 119903
1(119905) = 119886(120596
119909 120596119910 119905) and
1199032(119905) = 119887(120596
119909 120596119910 119905) respectively for a particular value of
(120596119909 120596119910) The pair of Gabor filters that provide these outputs
are the real and imaginary parts of119908(119909 119910)119890minus119895(120596119909119909+120596119910119910) It also
consists of a temporal high-pass 1198921(119905) and temporal low-
pass filter 1198922(119905) [9] The output of the elaborated Reichardt
detector follows the diagram in Figure 7 and can be expressedas
(1199032lowast 1198921) (119905) (119903
1lowast 1198922) (119905) minus (119903
1lowast 1198921) (119905) (119903
2lowast 1198922) (119905) (43)
The response can also be characterized by a second orderVolterra kernel We notice the striking similarity between
(43) and the numerator of (42) In fact the phase-basedmotion detector shares some properties with the Reichardtmotion detector For example it is straightforward to see thata single phase-basedmotion detector is tuned to the temporalfrequency of a moving sinusoidal grating
Since the motion energy detector is formally equivalentto an elaborated Reichardt motion detector [9] the structureof the motion energy detector with divisive normalization isalso similar to the phase-based motion detector
4 Exploratory Results
In this section we apply the phase-based motion detectionalgorithm on several video sequences and demonstrate itsefficiency and effectiveness in detecting local motion Themotion detection algorithm is compared to two well-knownbiological motion detectors namely the Reichardt motiondetector and the Barlow-Levick motion detector We thenshow that the detected local motion can be used in motionsegmentation tasks The effectiveness of the segmentationis compared to segmentation using motion informationobtained from a widely used optic flow based algorithmavailable in the literature [4]
Computational Intelligence and Neuroscience 11
(f) (e)
(b)
(a)
(d)
(c)
minus120587 minus1205872 0 1205872
10080604020
y (pi
xel)
100 15050x (pixel)
0020406081
15
10
5
l
01234567+
15
10
5
l
0510152025+
minus6
minus4
minus2
0
2
4
6
181716151413121110
9876543210
l
2 4 6 8 10 12 14 16 18 20 22 240k
minus6minus4minus20246
minus6minus4minus20246
120596x
minus120587 minus1205872 0 1205872
120596x
1205872
0
minus1205872
minus120587
120596y
1205872
0
minus1205872
minus120587
120596y
10 15 20 255k
10 20 255 15k
Figure 6 Evaluating the motion detection algorithm at time 1199050(see also Supplementary Video S1) (a) A still from a 156 times 112-pixel video at
time 1199050 (b) (119889120601
119896119897119889119905)(120596
119909 120596119910 1199050) for all 26times19 blocks each of size 32times32The blocks are concatenated to create a large ldquophase imagerdquo with their
relative neighbors kept (c) (1198891206011211
119889119905)(120596119909 120596119910 1199050) shown in the (120596
119909 120596119910) isin D2 space (d) (119889120601
236119889119905)(120596
119909 120596119910 1199050) shown in the (120596
119909 120596119910) isin D2
space (e) Phase Motion Indicator (PMI) for all blocks at 1199050 Each pixel (119896 119897) is color coded for PMI
119896119897 (f) max
(120596119909 120596119910)isinD2 |(119889120601119896119897119889119905)(120596
119909 120596119910 1199050)|
for all blocks at 1199050 Each pixel (119896 119897) is color coded for this maximum value of block (119896 119897)
g1(t)g2(t) g1(t) g2(t)
times times
minus++
SpatialGaborfilter
SpatialGaborfilter
r1(t) r2(t)
u(x y t)
Figure 7 Diagram of an elaborated Reichardt motion detector Itconsists of a quadrature pair of Gabor filters whose outputs 119903
1and 1199032
provide inputs to the high-pass filters 1198921(119905) and the low-pass filters
1198922(119905)
Table 1 Processing Speeds of the proposed algorithm the Reichardtmotion detector and the Barlow-Levick Motion detector
Screen size Processing capability (frames per second)Proposed Reichardt Barlow-Levick
320 times 240 420 790 800720 times 576 135 685 6901280 times 720 60 274 2931920 times 1080 27 191 194
41 Efficient Parallel Implementation The algorithm wasimplemented in PyCUDA [23] and tested on an NVIDIAGeForce GTX TITAN GPU All computations use singleprecision floating points The processing speeds of the algo-rithm for several screen sizes are listed in Table 1 Clearlythe proposed phase-based motion detection algorithm hasreal-time capability to process video even with full HighDefinition screen size
For comparison we implemented the Reichardt motiondetector and the Barlow-Levick motion detector Theirrespective diagrams are shown in Figure 8 Note that we
12 Computational Intelligence and Neuroscience
ukl(t) uk+1l(t)
g1(t)
g2(t)
g1(t)
g2(t)
+ minus+
timestimes
(a)
minus
minus minus
+
ukl(t) uk+1l(t)
g1(t)g1(t)
g2(t) g2(t)
++
+
(b)
Figure 8 (a) A Reichardt motion detector detecting motion between the two points 119906119896119897and 119906
119896+1119897in the blurred and downsampled video
(b) A Barlow-Levick motion detector detecting motion between the two points 119906119896119897and 119906
119896+1119897in the blurred and downsampled video 119892
1(119905)
denotes a high-pass filter and 1198922(119905) denotes a low-pass filter
moved the high-pass filter ℎ1(119905) to the front of the low-pass fil-
ter ℎ2(119905) This configuration provides a superior performance
to the one in Figure 7 For both the Reichardt and the Barlow-Levick detectors the videos are first blurred by a Gaussianfilter (with the same variance as the Gaussian window in (30)used for the phase-based motion detector) and subsampledat the center of each overlapping block in the phase-basedmotion detector The subsampled video then provides inputsto two 2D arrays of the circuits shown in Figure 8 one forthe horizontal direction and one for the vertical directionThe outputs of the horizontal and vertical motion circuitsform an array of motion vectors that indicate the strengthand direction of motion The three tested motion detectorshave the same number of outputs as a result The processingspeeds of the Reichardt motion detector and the Barlow-Levick motion detector both implemented in PyCUDA andtested on the same GPU are shown in Table 1
Note that the Reichardt motion detector and the Barlow-Levick motion detector are highly efficient due to the sim-plicity of their algorithmsThe phase-basedmotion detectionalgorithm however is a much more sophisticated algorithmand yet it can be implemented in real-time using parallelcomputing devices
The fast GPU implementation is based on the FFTand Matrix-Matrix multiplication It is expected that thoseoperations can be efficiently implemented in hardware forexample FPGA
42 Examples of Phased-BasedMotionDetection Weappliedour motion detection algorithm on video sequences of theChange Detection 2014 dataset [22] that did not exhibitcamera egomotion For these video sequences the standarddeviation of the Gaussian window functions was set to 4
pixels and the block size was chosen to be 32 times 32 pixelsThreshold and normalization parameters were kept the samewith the exception of the ldquothermal videordquo (in order to dealwith larger background noise levels see below) We alsotested the same video sequences using the Reichardt motiondetector and the Barlow-Levick motion detector For theReichardt detector the high-pass filters were chosen as firstorder filters with a time constant of 200 milliseconds and thelow-pass filters were chosen as first order filters with a timeconstant of 300 milliseconds (assuming that the frame rate is50 frames per second)Thresholdwas set to 2 For the Barlow-Levick motion detector the time constant of the first high-pass filters was set to 250 milliseconds The low-pass filterswere the same as in the Reichardt detectorThreshold was setto 2
The first video was taken from a highway surveillancecamera (ldquohighway videordquo) under good illumination con-ditions and high contrast The video had moderate noiseparticularly on the road surface The detected motion isshown in the top left panel of Figure 9 (see SupplementaryVideo S2 for full video) The phase-based motion detectionalgorithm captured both the moving cars and the tree leavesmoving (due to the wind) In the time interval between the9th and 10th second the camera was slightly moved left-and right-wards within 5 frames again possibly due to thewind Movement due to this shift was captured by themotiondetection algorithm and the algorithm was fast enough tocorrectly determine the direction of this movement In thisvideo we already noted that this algorithm suffers from theaperture problem For example in front of the van where along horizontal edge is present the detectedmotion ismostlypointing downwards In addition to moving downwards theedge is alsomoving to the left howeverThis is expected since
Computational Intelligence and Neuroscience 13
(a) (b) (c)
Figure 9 Top motion detection of the ldquohighway videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) Red arrows indicate detected motion and its direction Bottom the contrast of the video was artificiallyreduced by 5-fold and the mean was reduced to 35 of the original The red arrows are duplicated from the motion detection result on theoriginal video as in top Blue arrows are the result of motion detection on the video with reduced contrast If motion is detected both in theoriginal video and in the video with reduced contrast then the arrow is shown in magenta (as a mix of blue and red) (see SupplementaryVideo S2 for full video)
the algorithm only detects motion locally and does not takeinto account the overall shape of any object
For comparison the motion detection results for theReichardt motion detector and the Barlow-Levick motiondetector are shown in the top middle and top right panelof Figure 9 (see Supplementary Video S2 for full video) TheReichardt detector performed relatively well when vehiclesare moving faster but the direction it predicts for vehiclesmoving slower for example on the back of the image is notaccurate In addition motion is still detected in some partsof the screen where the vehicles have just passed by Thedetection result for the Barlow-Levick motion detector waspoorer In particular the response to OFF edge movement isalways the opposite to the actual movement direction
We then squeezed the range of the screen intensity from[0 1] to [02 04] resulting in a video with a lower meanluminance and lower contrast The motion detection resultson the low-contrast video are shown in the bottom 3 panelsof Figure 9 (see Supplementary Video S2 for full video) Forreference the motion detection results for the original videoare shown in red arrows Motion detected in the low-contrastvideo is shown in blue arrows if no motion is detected forthe block in the original video If motion is detected in boththe original and the low-contrast video the arrows are shownin magenta Figure 9 clearly shows that while the motiondetection performance is degraded for all three detectors thephase-basedmotion detector performed still quite well in thelow-contrast video detecting most of the moving vehicles
The other two detectors missed many of the blocks wheremotion was detected in the original movie
To quantify how well each detector works under differentcontrast conditions we computed the ratio of unthresholdedoutput values of eachmotion detector between lower contrastand full contrast video For example in the case of phase-basedmotion detector we computed for each block the ratiobetween the PMI index for the lower contrast video and thatfor the full contrast videoThe ratios are then averaged acrossall blocks where motion is detected in the full contrast videoThis average for different contrasts is shown in Figure 10 bythe blue curve In the ideal case when the normalizationconstant 120598 is 0 the phase detector should produce invariantresponses to videos with different contrast The curve shownhere is mainly due to a nonzero 120598The ratios for the Reichardtmotion detector and the Barlow-Levick motion detectorare computed similarly and are shown in red and yellowrespectively It is clear that the phase-based motion detectorhas a superior performance across a range of contrast valuesAt 20 contrast the phase-based detector still has 50 ofthe PMI index value for full contrast video As expectedthe response of Barlow-Levick motion detector is linear withrespect to contrast and the Reichardt motion detector has aquadratic relation to contrast For a fixed threshold value alarger ratio equates to amore consistent performance at lowercontrast
The second video was captured by a surveillance camerain a train station (ldquotrain station videordquo) The video was
14 Computational Intelligence and Neuroscience
Phase-basedReichardtBarlow-Levick
0
02
04
06
08
1
Ratio
40 60 80200 100Contrast ()
Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined
under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm
The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)
The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression
With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)
43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient
objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area
For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold
We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually
We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm
We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)
These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm
5 Discussion
Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes
Computational Intelligence and Neuroscience 15
(a) (b) (c)
Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)
(a) (b) (c)
Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)
by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to
models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]
We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in
16 Computational Intelligence and Neuroscience
(a) (b) (c)
Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)
Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)
phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in
Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)
modeling in vivo biological vision systems This is consistentwith other recent findings [15]
Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure
Computational Intelligence and Neuroscience 17
(a) (b)
Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)
(a) (b)
Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)
(a) (b)
Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)
Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase
itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third
18 Computational Intelligence and Neuroscience
(a) (b)
Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)
instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection
The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions
Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors
We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well
We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It
can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes
Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems
Appendix
The Derivative of the Local Phase
Let
119886 (120596119909 120596119910 119905)
= int
R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596
119909119909 + 120596119910119910) 119889119909 119889119910
119887 (120596119909 120596119910 119905)
= minusint
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
(A1)
and therefore
11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)
= 119886 (120596119909 120596119910 119905) + 119895119887 (120596
119909 120596119910 119905)
(A2)
The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596
119909
120596119910 119905) 119886(120596
119909 120596119910 119905)) and therefore
11988912060100
119889119905
=
119886 (119889119887119889119905) minus 119887 (119889119886119889119905)
[119886]2+ [119887]2
(A3)
Computational Intelligence and Neuroscience 19
Now
119889119886
119889119905
(120596119909 120596119910 119905)
= int
R2
120597119906
120597119905
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
(A4)
For a purely translational signal with instantaneous spatialshift given by (119904
119909(119905) 119904119910(119905)) within the support of the window
we have
119889119886
119889119905
(120596119909 120596119910 119905)
= minusint
R2
119889119904119909
119889119905
(119905)
120597119906
120597119909
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus int
R2
119889119904119910
119889119905
(119905)
120597119906
120597119910
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119909
119889119904119909
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119910
119889119904119910
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199033=1199033(120596119909120596119910119905)
+ 120596119909
119889119904119909
119889119905
119887
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199034=1199034(120596119909120596119910119905)
+ 120596119910
119889119904119910
119889119905
119887
(A5)
Similarly
119889119887
119889119905
(120596119909 120596119910 119905) = minus
119889119904119909
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199035=1199035(120596119909120596119910119905)
minus 120596119909
119889119904119909
119889119905
119886 minus
119889119904119910
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199036=1199036(120596119909120596119910119905)
minus 120596119910
119889119904119910
119889119905
119886
(A6)
Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to
11988912060100
119889119905
(120596119909 120596119910 119905) =
minus (119889119904119909119889119905) [119903
3119887 + 1199035119886 + 120596119909 [
119886]2+ 120596119909 [
119887]2] minus (119889119904
119910119889119905) [119903
4119887 + 1199036119886 + 120596119910 [
119886]2+ 120596119910 [
119887]2]
[119886]2+ [119887]2
= minus
119889119904119909
119889119905
120596119909minus
119889119904119910
119889119905
120596119910minus
119889119904119909
119889119905
sdot
1199033119887 + 1199035119886
[119886]2+ [119887]2minus
119889119904119910
119889119905
sdot
1199034119887 + 1199036119886
[119886]2+ [119887]2
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
v00(120596119909120596119910119905)
(A7)
Disclosure
The authorrsquos names are alphabetically listed
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgment
The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232
References
[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010
[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010
[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014
[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010
20 Computational Intelligence and Neuroscience
[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013
[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014
[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956
[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010
[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985
[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965
[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981
[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015
[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014
[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999
[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015
[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001
[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996
[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980
[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013
[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013
[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014
[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014
[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach
to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012
[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990
[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987
[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998
Submit your manuscripts athttpwwwhindawicom
Computer Games Technology
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Distributed Sensor Networks
International Journal of
Advances in
FuzzySystems
Hindawi Publishing Corporationhttpwwwhindawicom
Volume 2014
International Journal of
ReconfigurableComputing
Hindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Applied Computational Intelligence and Soft Computing
thinspAdvancesthinspinthinsp
Artificial Intelligence
HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014
Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Journal of
Computer Networks and Communications
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation
httpwwwhindawicom Volume 2014
Advances in
Multimedia
International Journal of
Biomedical Imaging
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
ArtificialNeural Systems
Advances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience
Industrial EngineeringJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Human-ComputerInteraction
Advances in
Computer EngineeringAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
6 Computational Intelligence and Neuroscience
us to reuse the same concept in detecting local motion Thiscan be achieved by restricting the domain of the visual fieldwhere the Fourier transform is applied
To be able to detect local motion we consider thelocal phase of 119906(119909 119910 119905) by taking the STFT with windowfunction 119908(119909 119910) Note that the STFT and its ubiquitousimplementation in DSP chips can be extensively used in anydimension For simplicity and without loss of generality weconsider the window to be centered at (0 0) The STFT isgiven by
int
R2119906 (119909 119910 119905) 119908 (119909 119910) 119890
minus119895(120596119909119909+120596119910119910)119889119909 119889119910
= 11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)
(27)
where by abuse of notation 11986000(120596119909 120596119910 119905) is the amplitude
and 12060100(120596119909 120596119910 119905) the local phase
Before we move on to the mathematical analysis we canintuitively explain the relation between the change in localphase and visual motion taking place across the windowsupport First if the stimulus undergoes a uniform changeof intensity or it changes proportionally over time due tolighting conditions for example the local phase does notchange since the phase is invariant with respect to intensityscaling Therefore the local phase does not change for suchnonmotion stimuli Second a rigid edge moving across thewindow support will induce a phase change
For a strictly translational signal within the windowsupport (footprint) for example
119906 (119909 119910 119905) = 119906 (119909 minus 119904119909(119905) 119910 minus 119904
119910(119905) 0)
for (119909 119910) isin supp (119908 (119909 119910))
(28)
where 119904119909(119905) and 119904
119910(119905) are as defined in (23) we have the
following result
Lemma 7 Consider the following
11988912060100
119889119905
(120596119909 120596119910 119905) = minus
119889119904119909(119905)
119889119905
120596119909minus
119889119904119910(119905)
119889119905
120596119910
+ v00(120596119909 120596119910 119905)
(29)
where by abuse of notation 12060100(120596119909 120596119910 119905) is the local phase
of 119906(119909 119910 119905) and 12060100(120596119909 120596119910 0) is the initial condition The
functional form of the term v00
= v00(120596119909 120596119910 119905) is provided
in the Appendix
Proof The derivation of (29) and the functional form of v00
are given in the Appendix
Remark 8 Wenotice that the derivative of the local phase hassimilar structure to that of the global phase but for the addedterm v
00Through simulations we observed that the first two
terms in (29) dominate over the last term for an ON or OFFmoving edge [3] For example Figure 3 shows the derivativeof the local phase given in (29) for an ON edge moving withvelocity (40 0) pixelssec
42
0minus2
minus4
100
d120601dt
120596yminus250minus200minus150minus100minus50
050
150
200250
31 2minus4 0minus1minus3 minus2 4
120596x
Figure 3 Derivative of the local phase for an ON edge moving inthe positive 119909 direction
Remark 9 Note that 12060100(120596119909 120596119910 119905) may not be differen-
tiable even if 119906(119909 119910 119905) is differentiable particularly when11986000(120596119909 120596119910 119905) = 0 For example the spatial phase can jump
from a positive value to zero when11986000(120596119909 120596119910 119905) diminishes
This also suggests that the instantaneous local spatial phaseis less informative about a region of a visual scene whenever11986000(120596119909 120596119910 119905) is close to zero Nevertheless the time deriva-
tive of the local phase can be approximated by applying ahigh-pass filter to 120601
00(120596119909 120596119910 119905)
322 The Block Structure for Computing the Local Phase Weconstruct Gaussian windows along the 119909 119910 dimensions TheGaussian windows are defined as
(T119896119897119908) (119909 119910) = 119890
minus((119909minus119909119896)2+(119909minus119910
119897)2)21205902
(30)
where 119909119896= 1198961198870 119910119897= 1198971198870 in which 119887
0isin Z+ is the distance
between two neighboring windows and 1 le 1198961198870le 119875119909 1 le
1198971198870le 119875119910 where 119875
119909 119875119910isin Z+ are the number of pixels of the
screen in 119909 and 119910 directions respectivelyWe then take the 2D Fourier transform of the windowed
video signal 119906(119909 119910 119905)(T119896119897119908)(119909 119910) and write in polar form
int
R2119906 (119909 119910 119905) (T
119896119897119908) (119909 119910) 119890
minus119895(120596119909(119909minus119909119896)+120596119910(119910minus119910119897))119889119909 119889119910
= 119860119896119897(120596119909 120596119910 119905) 119890119895120601119896119897(120596119909120596119910119905)
(31)
The above integral can be very efficiently evaluated usingthe 2D FFT in discrete domain defined on 119872 times 119872 blocksapproximating the footprint of the Gaussian windows Forexample the standard deviation of the Gaussian windows weuse in the examples in Section 4 is 4 pixels A block of 32times32
pixels (119872 = 32) is sufficient to cover the effective support(or footprint) of the Gaussian window At the same time thesize of the block is a power of 2 which is most suitable forFFT-based computationThe processing of each block (119896 119897) isindependent of all other blocks thereby parallelism is readilyachieved
Note that the size of the window is informed by thesize of the objects one is interested in locating Measure-ments of the local phase using smaller window functions
Computational Intelligence and Neuroscience 7
60
50
40
30
20
10
y (pi
xel)
01
02
03
04
05
06
07
08
09
1
20 30 40 50 6010x (pixel)
(a)
0
02
04
06
08
1
Mag
nitu
de (a
u)
10 20 30 40 50 60 700y (pixel)
(b)
Figure 4 Example of block structure (a) Four yellow disks show 4 Gaussian window functions with translation parameter 119896 = 3 119897 = 3
(top-left) 119896 = 3 119897 = 7 (bottom-left) 119896 = 7 119897 = 3 (top-right) and 119896 = 7 119897 = 7 (bottom-right) The red solid square shows a 32 times 32-pixel blockapproximating the Gaussian window with 119896 = 3 119897 = 3 and the green dashed square shows a 32 times 32-pixel block approximating the Gaussianwindow with 119896 = 3 119897 = 7 (b) Cross section of all 11 Gaussian windows with translation parameters 119896 = 7 119897 isin [0 10] The cross section istaken as indicated by the magenta line in (a) and the red and blue curve correspond to cross sections of the two Gaussian windows shown in(a) centered on the magenta line
are less robust to noise Larger windows would enhanceobject motion detection if the object size is comparable tothe window size However there would be an increasedlikelihood of independent movement of multiple objectswithin the same window which is not modeled here andthereby may not be robustly detected
Therefore for each block (119896 119897) we obtain 1198722 measure-
ments of the phase 120601119896119897(120596119909119898
120596119910119898
119905) at every time instant 119905with (120596
119909119898
120596119910119898
) isin D2 where
D2= (120596
119909119898
= 1198981205960 120596119910119899
= 1198991205960) 119898 119899 = minus1198722
minus 1198722 + 1 1198722 minus 1
(32)
with 1205960= 2120587119872 We then compute the temporal derivative
of the phase that is (119889120601119896119897119889119905)(120596
119909 120596119910 119905) for (120596
119909 120596119910) isin D2
We further illustrate an example of the block structurein Figure 4 Figure 4(a) shows an example of an image of64 times 64 pixels Four Gaussian windows are shown each witha standard deviation of 4 pixels The distance between thecenters of two neighboring Gaussian windows is 6 pixelsThe red solid square shows a 32 times 32-pixel block with 119896 =
3 119897 = 3 which encloses effective support of the Gaussianwindow on top-left (119896 = 0 119897 = 0 is the block with Gaussianwindow centered at pixel (1 1)) The green dashed squareshows another 32 times 32-pixel block with 119896 = 3 119897 = 7 Thetwo Gaussian windows on the right are associated with theblocks 119896 = 7 119897 = 3 and 119896 = 7 119897 = 7 respectively Crosssection of all Gaussian windows with 119896 = 7 119897 isin [0 10] that isthose centered on themagenta line are shown in Figure 4(b)The red and blue curve in Figure 4(b) correspond to the two
Gaussian windows shown in Figure 4(a) Figure 4(b) alsosuggests that some of the Gaussian windows are cut off on theboundaries This is however equivalent to assuming that thepixel values outside the boundary are always zero and it willnot significantly affect motion detection based on the changeof local phase
Since the phase and thereby the phase change is noisierwhen the local amplitude is low an additional denoisingstep can be employed to discount the measurements of(119889120601119896119897119889119905)(120596
119909 120596119910 119905) for low amplitude values 119860
119896119897(120596119909 120596119910 119905)
The denoising is given by
119889120601119896119897
119889119905
(120596119909 120596119910 119905)
sdot
119860119896119897(120596119909 120596119910 119905)
(11198722)sum(120596119909120596119910)isinD2 119860119896119897 (120596119909 120596119910 119905) + 120598
(33)
where 120598 gt 0 is a constant and (120596119909 120596119910) isin D2
33 The Phase-Based Detector We propose here a block FFTbased algorithm to detect motion using phase informationSuch an algorithm is due to its simplicity and parallelismhighly suitable for an in silico implementation
331 Radon Transform on the Change of Phases We exploitthe approximately linear structure of the phase derivative forblocks exhibiting motion by computing the Radon transformof (119889120601
119896119897119889119905)(120596
119909 120596119910 119905) over a circular bounded domain 119862 =
(120596119909 120596119910) | (120596119909 120596119910) isin D2 1205962
119909+ 1205962
119910lt 1205872
8 Computational Intelligence and Neuroscience
The Radon transform of the change of phase in thedomain 119862 is given by
(R119889120601119896119897
119889119905
) (120588 120579 119905) = int
R
119889120601119896119897
119889119905
(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588
sdot sin 120579 + 119904 sdot cos 120579 119905) 1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579
+ 119904 sdot cos 120579) 119889119904
(34)
where
1119862(120596119909 120596119910) =
1 if (120596119909 120596119910) isin 119862
0 otherwise(35)
The Radon transform (R(119889120601119896119897119889119905))(120588 120579 119905) evaluated at
a particular point (1205880 1205790 1199050) is essentially an integral
of (119889120601119896119897119889119905)(120596
119909 120596119910 1199050) along a line oriented at angle
1205872 + 1205790with the 120596
119909axis and at distance |120588
0| along the
(cos(1205790) sin(120579
0)) direction from (0 0)
If for a particular 119896 and 119897 we have (119889120601119896119897119889119905)(120596
119909 120596119910 119905) =
minusV119909(119905)120596119909minus V119910(119905)120596119910 we have
(R (119889120601119896119897119889119905)) (120588 120579 119905)
c (120588 120579)
= 120588 [minusV119909(119905) cos 120579 minus V
119910(119905) sin 120579]
(36)
where
c (120588 120579)
= int
R
1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579 + 119904 sdot cos 120579) 119889119904
(37)
c(120588 120579) is a correction term due to different length of lineintegrals for different values of (120588 120579) in the bounded domain119862
After computing the Radon transform of (119889120601119896119897119889119905)(120596
119909
120596119910 119905) for every block (119896 119897) at time 119905
0 we compute the Phase
Motion Indicator (PMI) defined as
PMI119896119897= max120579isin[0120587)
sum
120588
100381610038161003816100381610038161003816100381610038161003816
(R (119889120601119896119897119889119905)) (120588 120579 119905
0)
c (120588 120579)
100381610038161003816100381610038161003816100381610038161003816
(38)
If the PMI119896119897
is larger than a chosen threshold motion isdeemed to occur in block (119896 119897) at time 119905
0
Using the Radon transform makes it easier to separaterigid motion from noise Since the phase is quite sensitiveto noise particularly when the amplitude is very smallthe change of phase under noise may have comparablemagnitude to that due to motion as mentioned earlier Thechange of phase under noise however does not possess thestructure suggested by (29) in the (120596
119909 120596119910) domain Instead
it appears to be more randomly distributed Consequently
the PMI value is comparatively small for these blocks (see alsoSection 333)
Moreover the direction of motion for block (119896 119897) wheremotion is detected can be easily computed as
120579119896119897
= 120587(
sign (sum120588gt0
(R (119889120601119896119897119889119905)) (120588 120572
119896119897 1199050) c (120588 120572
119896119897)) + 1
2
)
+ 120572119896119897
(39)
where
120572119896119897= argmax120579isin[0120587)
sum
120588
100381610038161003816100381610038161003816100381610038161003816
(R (119889120601119896119897119889119905)) (120588 120579 119905
0)
c (120588 120579)
100381610038161003816100381610038161003816100381610038161003816
(40)
This follows from (36)
332 The Phase-Based Motion Detection Algorithm We for-mally summarize the above analysis as Algorithm 1 Figure 5shows a schematic diagram of the proposed phase-basedmotion detection algorithm
The algorithm is subdivided into two parts The first partcomputes local phase changes and the second part is thephase-based motion detector
In the first part the screen is divided into overlappingblocks For example the red green and blue blocks in theplane ldquodivide into overlapping blocksrdquo correspond to thesquares of the same color covering the video stream AGaussian window is then applied on each block followed bya 2D FFT operation that is used to extract the local phase Atemporal high-pass filter is then employed to extract phasechanges
In the second part the PMI is evaluated for each blockbased on the Radon transform of the local phase changes ineach block Motion is detected for blocks with PMI largerthan a preset threshold and the direction of motion iscomputed as in (39)
It is easy to notice that the algorithm can be highlyparallelized
333 Example We provide an illustrative example inFigure 6 showing how motion is detected using Algorithm 1The full video of this example can be found in SupplementaryVideo S1 see Supplementary Material available online athttpdxdoiorg10115520167915245 Figure 6(a) depicts astill from the ldquohighway videordquo in the Change Detection 2014dataset [22] evaluated at a particular time 119905
0 As suggested
by the algorithm the screen in Figure 6(a) is divided into26 times 19 overlapping blocks and the window functions areapplied to each block Local phases can then be extractedfrom the 2DFFT of eachwindowed block and the local phasechanges are obtained by temporal high-pass filtering Thephase change is shown in Figure 6(b) for all blocks with block(12 11) enlarged in Figure 6(c) and block (23 6) enlarged inFigure 6(d) (see also the plane ldquo2D FFT and extract phasechangerdquo in Figure 5) Note that at the time of the videoframe block (12 11) covers a part of the vehicle in motion
Computational Intelligence and Neuroscience 9
Input Visual stream or Video data 119906(119909 119910 119905)Output Detected MotionConstruct Gaussian window of size119872times119872 denote as 119908for 119905 = Δ119879 2Δ119879 119873Δ119879 do
Divide the screen of frame at time 119905 into multiple119872times119872 blocks 119906119896119897(119905) with overlaps
foreach block 119906119896119897(119905) do
Multiply pixel-wise the block 119906119896119897(119905) with the Gaussian window 119906
119896119897(119905)119908
Take119872times119872 2D FFT of 119906119896119897(119905)119908 denote phase as 120601
119896119897(120596119909 120596119910 119905) and amplitude as 119860
119896119897(120596119909 120596119910 119905)
foreach frequency (120596119909 120596119910) do
Highpass filter temporally 120601119896119897(120596119909 120596119910 119905) denote 1206011015840
119896119897(120596119909 120596119910 119905)
Optionally denoise by 1206011015840119896119897(120596119909 120596119910 119905) larr 120601
1015840
119896119897(120596119909 120596119910 119905)(119860
119896119897(120596119909 120596119910 119905)((1119872
2) sum(120596119909 120596119910)isinD
2 119860119896119897(120596119909 120596119910 119905) + 120598))
endCompute the radon transform (R120601
1015840
119896119897)(120588 120579 119905)
Compute PMI119896119897(119905) according to (38)
if PMI119896119897(119905) gt threshold then
Motion Detected at block (119896 119897) at time 119905Compute direction of motion according to (39)
endend
end
Algorithm 1 Phase-based motion detection algorithm using the FFT
y
120596x
120596y
y
t
120596x
120596y
x
y
x
y
x
x
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
Divided intooverlapping blocks
Apply window function
2D FFT andextract phase change
Radon transformand extract strength
and orientation of plane
Thresholding
Computephase
change
The phase-baseddetector
Figure 5 Schematic diagram of proposed phase-based motion detection algorithm
10 Computational Intelligence and Neuroscience
in the front and block (23 6) corresponds to an area of thehighway pavement where no motion occurs
Figure 6(f) depicts for each block (119896 119897) the maximumphase change over all (120596
119909 120596119910) isin D2 that is
max(120596119909120596119910)isinD2
10038161003816100381610038161003816100381610038161003816
119889120601119896119897
119889119905
(120596119909 120596119910 1199050)
10038161003816100381610038161003816100381610038161003816
(41)
We observe from the figure that for regions with lowamplitude such as the region depicting the road whenthe normalization constant is absent the derivative of thephase can be noisy For these blocks the maximum of|(119889120601119896119897119889119905)(120596
119909 120596119910 1199050)| over all (120596
119909 120596119910) isin D2 is comparable
to the maximum obtained for blocks that cover the vehiclesin motion
However (29) suggests that the local phase change frommultiple filter pairs centered at the same spatial position(119896 119897) can provide a constraint to robustly estimate motionand its direction Given the block structure employed in thecomputation of the local phase it is natural to utilize phasechange information from multiple sources
Indeed if for a particular block (119896 119897) (119889120601119896119897119889119905)(120596
119909
120596119910 119905) = minusV
119909(119905)120596119909minus V119910(119905)120596119910 then it is easy to see that (119889120601
119896119897
119889119905)(120596119909 120596119910 119905) will be zero on the line V
119909(119905)120596119909+ V119910(119905)120596119910= 0
and have opposite sign on either side of this line For examplein Figures 6(b) and 6(c) (119889120601
119896119897119889119905)(120596
119909 120596119910 1199050) clearly exhibits
this property for blocks that cover a vehicle in motion ThePMI is a tool to evaluate this property
Finally the PMIs for all blocks are shown compactlyin a heat map in Figure 6(e) The figure shows clearly thatthe blocks corresponding to the two moving vehicles have ahigh PMI value while the stationary background areas havea low PMI value allowing one to easily detect motion byemploying simple thresholding (see also the plane ldquoRadontransform and extract strength orientation of planerdquo inFigure 5) In addition the orientation ofmotion in each blockis readily observable even by inspection in Figure 6(b) by aline separating the yellow part and blue part in each blockFurther results about the direction of motion are presentedin Section 4
34 Relationship to Biological Motion Detectors A straight-forward way to implement local motion detectors is to applya complex-valued Gabor receptive field (5) to the video signal119906 and then take the derivative of the phase with respect totime or apply a high-pass filter on the phase to approximatethe derivative
We present here an alternate implementation withoutexplicitly computing the phaseThiswill elucidate the relationbetween the phase-based motion detector presented in theprevious section and some elementary motion detectionmodels used in biology such as theReichardtmotion detector[7] and motion energy detector [9]
Assuming that the local phase 12060100(120596119909 120596119910 119905) is differen-
tiable we have (see also the Appendix)
11988912060100(120596119909 120596119910 119905)
119889119905
=
(119889119887 (120596119909 120596119910 119905) 119889119905) 119886 (120596
119909 120596119910 119905) minus (119889119886 (120596
119909 120596119910 119905) 119889119905) 119887 (120596
119909 120596119910 119905)
[119886 (120596119909 120596119910 119905)]
2
+ [119887 (120596119909 120596119910 119905)]
2 (42)
where 119886(120596119909 120596119910 119905) and 119887(120596
119909 120596119910 119905) are respectively the real
and imaginary parts of 11986000(120596119909 120596119910 119905)11989011989512060100(120596119909120596119910119905)
We notice that the denominator of (42) is the square ofthe local amplitude of 119906 and the numerator is of the formof a second order Volterra kernel This suggests that thetime derivative of the local phase can be viewed as a secondorder Volterra kernel that processes two normalized spatiallyfiltered inputs V
1and V2
We consider an elaborated Reichardt motion detector asshown in Figure 7 It is equipped with a quadrature pair ofGabor filters whose outputs are 119903
1(119905) = 119886(120596
119909 120596119910 119905) and
1199032(119905) = 119887(120596
119909 120596119910 119905) respectively for a particular value of
(120596119909 120596119910) The pair of Gabor filters that provide these outputs
are the real and imaginary parts of119908(119909 119910)119890minus119895(120596119909119909+120596119910119910) It also
consists of a temporal high-pass 1198921(119905) and temporal low-
pass filter 1198922(119905) [9] The output of the elaborated Reichardt
detector follows the diagram in Figure 7 and can be expressedas
(1199032lowast 1198921) (119905) (119903
1lowast 1198922) (119905) minus (119903
1lowast 1198921) (119905) (119903
2lowast 1198922) (119905) (43)
The response can also be characterized by a second orderVolterra kernel We notice the striking similarity between
(43) and the numerator of (42) In fact the phase-basedmotion detector shares some properties with the Reichardtmotion detector For example it is straightforward to see thata single phase-basedmotion detector is tuned to the temporalfrequency of a moving sinusoidal grating
Since the motion energy detector is formally equivalentto an elaborated Reichardt motion detector [9] the structureof the motion energy detector with divisive normalization isalso similar to the phase-based motion detector
4 Exploratory Results
In this section we apply the phase-based motion detectionalgorithm on several video sequences and demonstrate itsefficiency and effectiveness in detecting local motion Themotion detection algorithm is compared to two well-knownbiological motion detectors namely the Reichardt motiondetector and the Barlow-Levick motion detector We thenshow that the detected local motion can be used in motionsegmentation tasks The effectiveness of the segmentationis compared to segmentation using motion informationobtained from a widely used optic flow based algorithmavailable in the literature [4]
Computational Intelligence and Neuroscience 11
(f) (e)
(b)
(a)
(d)
(c)
minus120587 minus1205872 0 1205872
10080604020
y (pi
xel)
100 15050x (pixel)
0020406081
15
10
5
l
01234567+
15
10
5
l
0510152025+
minus6
minus4
minus2
0
2
4
6
181716151413121110
9876543210
l
2 4 6 8 10 12 14 16 18 20 22 240k
minus6minus4minus20246
minus6minus4minus20246
120596x
minus120587 minus1205872 0 1205872
120596x
1205872
0
minus1205872
minus120587
120596y
1205872
0
minus1205872
minus120587
120596y
10 15 20 255k
10 20 255 15k
Figure 6 Evaluating the motion detection algorithm at time 1199050(see also Supplementary Video S1) (a) A still from a 156 times 112-pixel video at
time 1199050 (b) (119889120601
119896119897119889119905)(120596
119909 120596119910 1199050) for all 26times19 blocks each of size 32times32The blocks are concatenated to create a large ldquophase imagerdquo with their
relative neighbors kept (c) (1198891206011211
119889119905)(120596119909 120596119910 1199050) shown in the (120596
119909 120596119910) isin D2 space (d) (119889120601
236119889119905)(120596
119909 120596119910 1199050) shown in the (120596
119909 120596119910) isin D2
space (e) Phase Motion Indicator (PMI) for all blocks at 1199050 Each pixel (119896 119897) is color coded for PMI
119896119897 (f) max
(120596119909 120596119910)isinD2 |(119889120601119896119897119889119905)(120596
119909 120596119910 1199050)|
for all blocks at 1199050 Each pixel (119896 119897) is color coded for this maximum value of block (119896 119897)
g1(t)g2(t) g1(t) g2(t)
times times
minus++
SpatialGaborfilter
SpatialGaborfilter
r1(t) r2(t)
u(x y t)
Figure 7 Diagram of an elaborated Reichardt motion detector Itconsists of a quadrature pair of Gabor filters whose outputs 119903
1and 1199032
provide inputs to the high-pass filters 1198921(119905) and the low-pass filters
1198922(119905)
Table 1 Processing Speeds of the proposed algorithm the Reichardtmotion detector and the Barlow-Levick Motion detector
Screen size Processing capability (frames per second)Proposed Reichardt Barlow-Levick
320 times 240 420 790 800720 times 576 135 685 6901280 times 720 60 274 2931920 times 1080 27 191 194
41 Efficient Parallel Implementation The algorithm wasimplemented in PyCUDA [23] and tested on an NVIDIAGeForce GTX TITAN GPU All computations use singleprecision floating points The processing speeds of the algo-rithm for several screen sizes are listed in Table 1 Clearlythe proposed phase-based motion detection algorithm hasreal-time capability to process video even with full HighDefinition screen size
For comparison we implemented the Reichardt motiondetector and the Barlow-Levick motion detector Theirrespective diagrams are shown in Figure 8 Note that we
12 Computational Intelligence and Neuroscience
ukl(t) uk+1l(t)
g1(t)
g2(t)
g1(t)
g2(t)
+ minus+
timestimes
(a)
minus
minus minus
+
ukl(t) uk+1l(t)
g1(t)g1(t)
g2(t) g2(t)
++
+
(b)
Figure 8 (a) A Reichardt motion detector detecting motion between the two points 119906119896119897and 119906
119896+1119897in the blurred and downsampled video
(b) A Barlow-Levick motion detector detecting motion between the two points 119906119896119897and 119906
119896+1119897in the blurred and downsampled video 119892
1(119905)
denotes a high-pass filter and 1198922(119905) denotes a low-pass filter
moved the high-pass filter ℎ1(119905) to the front of the low-pass fil-
ter ℎ2(119905) This configuration provides a superior performance
to the one in Figure 7 For both the Reichardt and the Barlow-Levick detectors the videos are first blurred by a Gaussianfilter (with the same variance as the Gaussian window in (30)used for the phase-based motion detector) and subsampledat the center of each overlapping block in the phase-basedmotion detector The subsampled video then provides inputsto two 2D arrays of the circuits shown in Figure 8 one forthe horizontal direction and one for the vertical directionThe outputs of the horizontal and vertical motion circuitsform an array of motion vectors that indicate the strengthand direction of motion The three tested motion detectorshave the same number of outputs as a result The processingspeeds of the Reichardt motion detector and the Barlow-Levick motion detector both implemented in PyCUDA andtested on the same GPU are shown in Table 1
Note that the Reichardt motion detector and the Barlow-Levick motion detector are highly efficient due to the sim-plicity of their algorithmsThe phase-basedmotion detectionalgorithm however is a much more sophisticated algorithmand yet it can be implemented in real-time using parallelcomputing devices
The fast GPU implementation is based on the FFTand Matrix-Matrix multiplication It is expected that thoseoperations can be efficiently implemented in hardware forexample FPGA
42 Examples of Phased-BasedMotionDetection Weappliedour motion detection algorithm on video sequences of theChange Detection 2014 dataset [22] that did not exhibitcamera egomotion For these video sequences the standarddeviation of the Gaussian window functions was set to 4
pixels and the block size was chosen to be 32 times 32 pixelsThreshold and normalization parameters were kept the samewith the exception of the ldquothermal videordquo (in order to dealwith larger background noise levels see below) We alsotested the same video sequences using the Reichardt motiondetector and the Barlow-Levick motion detector For theReichardt detector the high-pass filters were chosen as firstorder filters with a time constant of 200 milliseconds and thelow-pass filters were chosen as first order filters with a timeconstant of 300 milliseconds (assuming that the frame rate is50 frames per second)Thresholdwas set to 2 For the Barlow-Levick motion detector the time constant of the first high-pass filters was set to 250 milliseconds The low-pass filterswere the same as in the Reichardt detectorThreshold was setto 2
The first video was taken from a highway surveillancecamera (ldquohighway videordquo) under good illumination con-ditions and high contrast The video had moderate noiseparticularly on the road surface The detected motion isshown in the top left panel of Figure 9 (see SupplementaryVideo S2 for full video) The phase-based motion detectionalgorithm captured both the moving cars and the tree leavesmoving (due to the wind) In the time interval between the9th and 10th second the camera was slightly moved left-and right-wards within 5 frames again possibly due to thewind Movement due to this shift was captured by themotiondetection algorithm and the algorithm was fast enough tocorrectly determine the direction of this movement In thisvideo we already noted that this algorithm suffers from theaperture problem For example in front of the van where along horizontal edge is present the detectedmotion ismostlypointing downwards In addition to moving downwards theedge is alsomoving to the left howeverThis is expected since
Computational Intelligence and Neuroscience 13
(a) (b) (c)
Figure 9 Top motion detection of the ldquohighway videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) Red arrows indicate detected motion and its direction Bottom the contrast of the video was artificiallyreduced by 5-fold and the mean was reduced to 35 of the original The red arrows are duplicated from the motion detection result on theoriginal video as in top Blue arrows are the result of motion detection on the video with reduced contrast If motion is detected both in theoriginal video and in the video with reduced contrast then the arrow is shown in magenta (as a mix of blue and red) (see SupplementaryVideo S2 for full video)
the algorithm only detects motion locally and does not takeinto account the overall shape of any object
For comparison the motion detection results for theReichardt motion detector and the Barlow-Levick motiondetector are shown in the top middle and top right panelof Figure 9 (see Supplementary Video S2 for full video) TheReichardt detector performed relatively well when vehiclesare moving faster but the direction it predicts for vehiclesmoving slower for example on the back of the image is notaccurate In addition motion is still detected in some partsof the screen where the vehicles have just passed by Thedetection result for the Barlow-Levick motion detector waspoorer In particular the response to OFF edge movement isalways the opposite to the actual movement direction
We then squeezed the range of the screen intensity from[0 1] to [02 04] resulting in a video with a lower meanluminance and lower contrast The motion detection resultson the low-contrast video are shown in the bottom 3 panelsof Figure 9 (see Supplementary Video S2 for full video) Forreference the motion detection results for the original videoare shown in red arrows Motion detected in the low-contrastvideo is shown in blue arrows if no motion is detected forthe block in the original video If motion is detected in boththe original and the low-contrast video the arrows are shownin magenta Figure 9 clearly shows that while the motiondetection performance is degraded for all three detectors thephase-basedmotion detector performed still quite well in thelow-contrast video detecting most of the moving vehicles
The other two detectors missed many of the blocks wheremotion was detected in the original movie
To quantify how well each detector works under differentcontrast conditions we computed the ratio of unthresholdedoutput values of eachmotion detector between lower contrastand full contrast video For example in the case of phase-basedmotion detector we computed for each block the ratiobetween the PMI index for the lower contrast video and thatfor the full contrast videoThe ratios are then averaged acrossall blocks where motion is detected in the full contrast videoThis average for different contrasts is shown in Figure 10 bythe blue curve In the ideal case when the normalizationconstant 120598 is 0 the phase detector should produce invariantresponses to videos with different contrast The curve shownhere is mainly due to a nonzero 120598The ratios for the Reichardtmotion detector and the Barlow-Levick motion detectorare computed similarly and are shown in red and yellowrespectively It is clear that the phase-based motion detectorhas a superior performance across a range of contrast valuesAt 20 contrast the phase-based detector still has 50 ofthe PMI index value for full contrast video As expectedthe response of Barlow-Levick motion detector is linear withrespect to contrast and the Reichardt motion detector has aquadratic relation to contrast For a fixed threshold value alarger ratio equates to amore consistent performance at lowercontrast
The second video was captured by a surveillance camerain a train station (ldquotrain station videordquo) The video was
14 Computational Intelligence and Neuroscience
Phase-basedReichardtBarlow-Levick
0
02
04
06
08
1
Ratio
40 60 80200 100Contrast ()
Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined
under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm
The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)
The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression
With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)
43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient
objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area
For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold
We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually
We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm
We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)
These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm
5 Discussion
Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes
Computational Intelligence and Neuroscience 15
(a) (b) (c)
Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)
(a) (b) (c)
Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)
by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to
models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]
We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in
16 Computational Intelligence and Neuroscience
(a) (b) (c)
Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)
Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)
phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in
Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)
modeling in vivo biological vision systems This is consistentwith other recent findings [15]
Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure
Computational Intelligence and Neuroscience 17
(a) (b)
Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)
(a) (b)
Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)
(a) (b)
Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)
Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase
itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third
18 Computational Intelligence and Neuroscience
(a) (b)
Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)
instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection
The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions
Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors
We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well
We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It
can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes
Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems
Appendix
The Derivative of the Local Phase
Let
119886 (120596119909 120596119910 119905)
= int
R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596
119909119909 + 120596119910119910) 119889119909 119889119910
119887 (120596119909 120596119910 119905)
= minusint
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
(A1)
and therefore
11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)
= 119886 (120596119909 120596119910 119905) + 119895119887 (120596
119909 120596119910 119905)
(A2)
The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596
119909
120596119910 119905) 119886(120596
119909 120596119910 119905)) and therefore
11988912060100
119889119905
=
119886 (119889119887119889119905) minus 119887 (119889119886119889119905)
[119886]2+ [119887]2
(A3)
Computational Intelligence and Neuroscience 19
Now
119889119886
119889119905
(120596119909 120596119910 119905)
= int
R2
120597119906
120597119905
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
(A4)
For a purely translational signal with instantaneous spatialshift given by (119904
119909(119905) 119904119910(119905)) within the support of the window
we have
119889119886
119889119905
(120596119909 120596119910 119905)
= minusint
R2
119889119904119909
119889119905
(119905)
120597119906
120597119909
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus int
R2
119889119904119910
119889119905
(119905)
120597119906
120597119910
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119909
119889119904119909
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119910
119889119904119910
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199033=1199033(120596119909120596119910119905)
+ 120596119909
119889119904119909
119889119905
119887
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199034=1199034(120596119909120596119910119905)
+ 120596119910
119889119904119910
119889119905
119887
(A5)
Similarly
119889119887
119889119905
(120596119909 120596119910 119905) = minus
119889119904119909
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199035=1199035(120596119909120596119910119905)
minus 120596119909
119889119904119909
119889119905
119886 minus
119889119904119910
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199036=1199036(120596119909120596119910119905)
minus 120596119910
119889119904119910
119889119905
119886
(A6)
Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to
11988912060100
119889119905
(120596119909 120596119910 119905) =
minus (119889119904119909119889119905) [119903
3119887 + 1199035119886 + 120596119909 [
119886]2+ 120596119909 [
119887]2] minus (119889119904
119910119889119905) [119903
4119887 + 1199036119886 + 120596119910 [
119886]2+ 120596119910 [
119887]2]
[119886]2+ [119887]2
= minus
119889119904119909
119889119905
120596119909minus
119889119904119910
119889119905
120596119910minus
119889119904119909
119889119905
sdot
1199033119887 + 1199035119886
[119886]2+ [119887]2minus
119889119904119910
119889119905
sdot
1199034119887 + 1199036119886
[119886]2+ [119887]2
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
v00(120596119909120596119910119905)
(A7)
Disclosure
The authorrsquos names are alphabetically listed
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgment
The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232
References
[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010
[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010
[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014
[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010
20 Computational Intelligence and Neuroscience
[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013
[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014
[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956
[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010
[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985
[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965
[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981
[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015
[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014
[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999
[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015
[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001
[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996
[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980
[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013
[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013
[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014
[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014
[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach
to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012
[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990
[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987
[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998
Submit your manuscripts athttpwwwhindawicom
Computer Games Technology
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Distributed Sensor Networks
International Journal of
Advances in
FuzzySystems
Hindawi Publishing Corporationhttpwwwhindawicom
Volume 2014
International Journal of
ReconfigurableComputing
Hindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Applied Computational Intelligence and Soft Computing
thinspAdvancesthinspinthinsp
Artificial Intelligence
HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014
Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Journal of
Computer Networks and Communications
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation
httpwwwhindawicom Volume 2014
Advances in
Multimedia
International Journal of
Biomedical Imaging
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
ArtificialNeural Systems
Advances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience
Industrial EngineeringJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Human-ComputerInteraction
Advances in
Computer EngineeringAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience 7
60
50
40
30
20
10
y (pi
xel)
01
02
03
04
05
06
07
08
09
1
20 30 40 50 6010x (pixel)
(a)
0
02
04
06
08
1
Mag
nitu
de (a
u)
10 20 30 40 50 60 700y (pixel)
(b)
Figure 4 Example of block structure (a) Four yellow disks show 4 Gaussian window functions with translation parameter 119896 = 3 119897 = 3
(top-left) 119896 = 3 119897 = 7 (bottom-left) 119896 = 7 119897 = 3 (top-right) and 119896 = 7 119897 = 7 (bottom-right) The red solid square shows a 32 times 32-pixel blockapproximating the Gaussian window with 119896 = 3 119897 = 3 and the green dashed square shows a 32 times 32-pixel block approximating the Gaussianwindow with 119896 = 3 119897 = 7 (b) Cross section of all 11 Gaussian windows with translation parameters 119896 = 7 119897 isin [0 10] The cross section istaken as indicated by the magenta line in (a) and the red and blue curve correspond to cross sections of the two Gaussian windows shown in(a) centered on the magenta line
are less robust to noise Larger windows would enhanceobject motion detection if the object size is comparable tothe window size However there would be an increasedlikelihood of independent movement of multiple objectswithin the same window which is not modeled here andthereby may not be robustly detected
Therefore for each block (119896 119897) we obtain 1198722 measure-
ments of the phase 120601119896119897(120596119909119898
120596119910119898
119905) at every time instant 119905with (120596
119909119898
120596119910119898
) isin D2 where
D2= (120596
119909119898
= 1198981205960 120596119910119899
= 1198991205960) 119898 119899 = minus1198722
minus 1198722 + 1 1198722 minus 1
(32)
with 1205960= 2120587119872 We then compute the temporal derivative
of the phase that is (119889120601119896119897119889119905)(120596
119909 120596119910 119905) for (120596
119909 120596119910) isin D2
We further illustrate an example of the block structurein Figure 4 Figure 4(a) shows an example of an image of64 times 64 pixels Four Gaussian windows are shown each witha standard deviation of 4 pixels The distance between thecenters of two neighboring Gaussian windows is 6 pixelsThe red solid square shows a 32 times 32-pixel block with 119896 =
3 119897 = 3 which encloses effective support of the Gaussianwindow on top-left (119896 = 0 119897 = 0 is the block with Gaussianwindow centered at pixel (1 1)) The green dashed squareshows another 32 times 32-pixel block with 119896 = 3 119897 = 7 Thetwo Gaussian windows on the right are associated with theblocks 119896 = 7 119897 = 3 and 119896 = 7 119897 = 7 respectively Crosssection of all Gaussian windows with 119896 = 7 119897 isin [0 10] that isthose centered on themagenta line are shown in Figure 4(b)The red and blue curve in Figure 4(b) correspond to the two
Gaussian windows shown in Figure 4(a) Figure 4(b) alsosuggests that some of the Gaussian windows are cut off on theboundaries This is however equivalent to assuming that thepixel values outside the boundary are always zero and it willnot significantly affect motion detection based on the changeof local phase
Since the phase and thereby the phase change is noisierwhen the local amplitude is low an additional denoisingstep can be employed to discount the measurements of(119889120601119896119897119889119905)(120596
119909 120596119910 119905) for low amplitude values 119860
119896119897(120596119909 120596119910 119905)
The denoising is given by
119889120601119896119897
119889119905
(120596119909 120596119910 119905)
sdot
119860119896119897(120596119909 120596119910 119905)
(11198722)sum(120596119909120596119910)isinD2 119860119896119897 (120596119909 120596119910 119905) + 120598
(33)
where 120598 gt 0 is a constant and (120596119909 120596119910) isin D2
33 The Phase-Based Detector We propose here a block FFTbased algorithm to detect motion using phase informationSuch an algorithm is due to its simplicity and parallelismhighly suitable for an in silico implementation
331 Radon Transform on the Change of Phases We exploitthe approximately linear structure of the phase derivative forblocks exhibiting motion by computing the Radon transformof (119889120601
119896119897119889119905)(120596
119909 120596119910 119905) over a circular bounded domain 119862 =
(120596119909 120596119910) | (120596119909 120596119910) isin D2 1205962
119909+ 1205962
119910lt 1205872
8 Computational Intelligence and Neuroscience
The Radon transform of the change of phase in thedomain 119862 is given by
(R119889120601119896119897
119889119905
) (120588 120579 119905) = int
R
119889120601119896119897
119889119905
(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588
sdot sin 120579 + 119904 sdot cos 120579 119905) 1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579
+ 119904 sdot cos 120579) 119889119904
(34)
where
1119862(120596119909 120596119910) =
1 if (120596119909 120596119910) isin 119862
0 otherwise(35)
The Radon transform (R(119889120601119896119897119889119905))(120588 120579 119905) evaluated at
a particular point (1205880 1205790 1199050) is essentially an integral
of (119889120601119896119897119889119905)(120596
119909 120596119910 1199050) along a line oriented at angle
1205872 + 1205790with the 120596
119909axis and at distance |120588
0| along the
(cos(1205790) sin(120579
0)) direction from (0 0)
If for a particular 119896 and 119897 we have (119889120601119896119897119889119905)(120596
119909 120596119910 119905) =
minusV119909(119905)120596119909minus V119910(119905)120596119910 we have
(R (119889120601119896119897119889119905)) (120588 120579 119905)
c (120588 120579)
= 120588 [minusV119909(119905) cos 120579 minus V
119910(119905) sin 120579]
(36)
where
c (120588 120579)
= int
R
1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579 + 119904 sdot cos 120579) 119889119904
(37)
c(120588 120579) is a correction term due to different length of lineintegrals for different values of (120588 120579) in the bounded domain119862
After computing the Radon transform of (119889120601119896119897119889119905)(120596
119909
120596119910 119905) for every block (119896 119897) at time 119905
0 we compute the Phase
Motion Indicator (PMI) defined as
PMI119896119897= max120579isin[0120587)
sum
120588
100381610038161003816100381610038161003816100381610038161003816
(R (119889120601119896119897119889119905)) (120588 120579 119905
0)
c (120588 120579)
100381610038161003816100381610038161003816100381610038161003816
(38)
If the PMI119896119897
is larger than a chosen threshold motion isdeemed to occur in block (119896 119897) at time 119905
0
Using the Radon transform makes it easier to separaterigid motion from noise Since the phase is quite sensitiveto noise particularly when the amplitude is very smallthe change of phase under noise may have comparablemagnitude to that due to motion as mentioned earlier Thechange of phase under noise however does not possess thestructure suggested by (29) in the (120596
119909 120596119910) domain Instead
it appears to be more randomly distributed Consequently
the PMI value is comparatively small for these blocks (see alsoSection 333)
Moreover the direction of motion for block (119896 119897) wheremotion is detected can be easily computed as
120579119896119897
= 120587(
sign (sum120588gt0
(R (119889120601119896119897119889119905)) (120588 120572
119896119897 1199050) c (120588 120572
119896119897)) + 1
2
)
+ 120572119896119897
(39)
where
120572119896119897= argmax120579isin[0120587)
sum
120588
100381610038161003816100381610038161003816100381610038161003816
(R (119889120601119896119897119889119905)) (120588 120579 119905
0)
c (120588 120579)
100381610038161003816100381610038161003816100381610038161003816
(40)
This follows from (36)
332 The Phase-Based Motion Detection Algorithm We for-mally summarize the above analysis as Algorithm 1 Figure 5shows a schematic diagram of the proposed phase-basedmotion detection algorithm
The algorithm is subdivided into two parts The first partcomputes local phase changes and the second part is thephase-based motion detector
In the first part the screen is divided into overlappingblocks For example the red green and blue blocks in theplane ldquodivide into overlapping blocksrdquo correspond to thesquares of the same color covering the video stream AGaussian window is then applied on each block followed bya 2D FFT operation that is used to extract the local phase Atemporal high-pass filter is then employed to extract phasechanges
In the second part the PMI is evaluated for each blockbased on the Radon transform of the local phase changes ineach block Motion is detected for blocks with PMI largerthan a preset threshold and the direction of motion iscomputed as in (39)
It is easy to notice that the algorithm can be highlyparallelized
333 Example We provide an illustrative example inFigure 6 showing how motion is detected using Algorithm 1The full video of this example can be found in SupplementaryVideo S1 see Supplementary Material available online athttpdxdoiorg10115520167915245 Figure 6(a) depicts astill from the ldquohighway videordquo in the Change Detection 2014dataset [22] evaluated at a particular time 119905
0 As suggested
by the algorithm the screen in Figure 6(a) is divided into26 times 19 overlapping blocks and the window functions areapplied to each block Local phases can then be extractedfrom the 2DFFT of eachwindowed block and the local phasechanges are obtained by temporal high-pass filtering Thephase change is shown in Figure 6(b) for all blocks with block(12 11) enlarged in Figure 6(c) and block (23 6) enlarged inFigure 6(d) (see also the plane ldquo2D FFT and extract phasechangerdquo in Figure 5) Note that at the time of the videoframe block (12 11) covers a part of the vehicle in motion
Computational Intelligence and Neuroscience 9
Input Visual stream or Video data 119906(119909 119910 119905)Output Detected MotionConstruct Gaussian window of size119872times119872 denote as 119908for 119905 = Δ119879 2Δ119879 119873Δ119879 do
Divide the screen of frame at time 119905 into multiple119872times119872 blocks 119906119896119897(119905) with overlaps
foreach block 119906119896119897(119905) do
Multiply pixel-wise the block 119906119896119897(119905) with the Gaussian window 119906
119896119897(119905)119908
Take119872times119872 2D FFT of 119906119896119897(119905)119908 denote phase as 120601
119896119897(120596119909 120596119910 119905) and amplitude as 119860
119896119897(120596119909 120596119910 119905)
foreach frequency (120596119909 120596119910) do
Highpass filter temporally 120601119896119897(120596119909 120596119910 119905) denote 1206011015840
119896119897(120596119909 120596119910 119905)
Optionally denoise by 1206011015840119896119897(120596119909 120596119910 119905) larr 120601
1015840
119896119897(120596119909 120596119910 119905)(119860
119896119897(120596119909 120596119910 119905)((1119872
2) sum(120596119909 120596119910)isinD
2 119860119896119897(120596119909 120596119910 119905) + 120598))
endCompute the radon transform (R120601
1015840
119896119897)(120588 120579 119905)
Compute PMI119896119897(119905) according to (38)
if PMI119896119897(119905) gt threshold then
Motion Detected at block (119896 119897) at time 119905Compute direction of motion according to (39)
endend
end
Algorithm 1 Phase-based motion detection algorithm using the FFT
y
120596x
120596y
y
t
120596x
120596y
x
y
x
y
x
x
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
Divided intooverlapping blocks
Apply window function
2D FFT andextract phase change
Radon transformand extract strength
and orientation of plane
Thresholding
Computephase
change
The phase-baseddetector
Figure 5 Schematic diagram of proposed phase-based motion detection algorithm
10 Computational Intelligence and Neuroscience
in the front and block (23 6) corresponds to an area of thehighway pavement where no motion occurs
Figure 6(f) depicts for each block (119896 119897) the maximumphase change over all (120596
119909 120596119910) isin D2 that is
max(120596119909120596119910)isinD2
10038161003816100381610038161003816100381610038161003816
119889120601119896119897
119889119905
(120596119909 120596119910 1199050)
10038161003816100381610038161003816100381610038161003816
(41)
We observe from the figure that for regions with lowamplitude such as the region depicting the road whenthe normalization constant is absent the derivative of thephase can be noisy For these blocks the maximum of|(119889120601119896119897119889119905)(120596
119909 120596119910 1199050)| over all (120596
119909 120596119910) isin D2 is comparable
to the maximum obtained for blocks that cover the vehiclesin motion
However (29) suggests that the local phase change frommultiple filter pairs centered at the same spatial position(119896 119897) can provide a constraint to robustly estimate motionand its direction Given the block structure employed in thecomputation of the local phase it is natural to utilize phasechange information from multiple sources
Indeed if for a particular block (119896 119897) (119889120601119896119897119889119905)(120596
119909
120596119910 119905) = minusV
119909(119905)120596119909minus V119910(119905)120596119910 then it is easy to see that (119889120601
119896119897
119889119905)(120596119909 120596119910 119905) will be zero on the line V
119909(119905)120596119909+ V119910(119905)120596119910= 0
and have opposite sign on either side of this line For examplein Figures 6(b) and 6(c) (119889120601
119896119897119889119905)(120596
119909 120596119910 1199050) clearly exhibits
this property for blocks that cover a vehicle in motion ThePMI is a tool to evaluate this property
Finally the PMIs for all blocks are shown compactlyin a heat map in Figure 6(e) The figure shows clearly thatthe blocks corresponding to the two moving vehicles have ahigh PMI value while the stationary background areas havea low PMI value allowing one to easily detect motion byemploying simple thresholding (see also the plane ldquoRadontransform and extract strength orientation of planerdquo inFigure 5) In addition the orientation ofmotion in each blockis readily observable even by inspection in Figure 6(b) by aline separating the yellow part and blue part in each blockFurther results about the direction of motion are presentedin Section 4
34 Relationship to Biological Motion Detectors A straight-forward way to implement local motion detectors is to applya complex-valued Gabor receptive field (5) to the video signal119906 and then take the derivative of the phase with respect totime or apply a high-pass filter on the phase to approximatethe derivative
We present here an alternate implementation withoutexplicitly computing the phaseThiswill elucidate the relationbetween the phase-based motion detector presented in theprevious section and some elementary motion detectionmodels used in biology such as theReichardtmotion detector[7] and motion energy detector [9]
Assuming that the local phase 12060100(120596119909 120596119910 119905) is differen-
tiable we have (see also the Appendix)
11988912060100(120596119909 120596119910 119905)
119889119905
=
(119889119887 (120596119909 120596119910 119905) 119889119905) 119886 (120596
119909 120596119910 119905) minus (119889119886 (120596
119909 120596119910 119905) 119889119905) 119887 (120596
119909 120596119910 119905)
[119886 (120596119909 120596119910 119905)]
2
+ [119887 (120596119909 120596119910 119905)]
2 (42)
where 119886(120596119909 120596119910 119905) and 119887(120596
119909 120596119910 119905) are respectively the real
and imaginary parts of 11986000(120596119909 120596119910 119905)11989011989512060100(120596119909120596119910119905)
We notice that the denominator of (42) is the square ofthe local amplitude of 119906 and the numerator is of the formof a second order Volterra kernel This suggests that thetime derivative of the local phase can be viewed as a secondorder Volterra kernel that processes two normalized spatiallyfiltered inputs V
1and V2
We consider an elaborated Reichardt motion detector asshown in Figure 7 It is equipped with a quadrature pair ofGabor filters whose outputs are 119903
1(119905) = 119886(120596
119909 120596119910 119905) and
1199032(119905) = 119887(120596
119909 120596119910 119905) respectively for a particular value of
(120596119909 120596119910) The pair of Gabor filters that provide these outputs
are the real and imaginary parts of119908(119909 119910)119890minus119895(120596119909119909+120596119910119910) It also
consists of a temporal high-pass 1198921(119905) and temporal low-
pass filter 1198922(119905) [9] The output of the elaborated Reichardt
detector follows the diagram in Figure 7 and can be expressedas
(1199032lowast 1198921) (119905) (119903
1lowast 1198922) (119905) minus (119903
1lowast 1198921) (119905) (119903
2lowast 1198922) (119905) (43)
The response can also be characterized by a second orderVolterra kernel We notice the striking similarity between
(43) and the numerator of (42) In fact the phase-basedmotion detector shares some properties with the Reichardtmotion detector For example it is straightforward to see thata single phase-basedmotion detector is tuned to the temporalfrequency of a moving sinusoidal grating
Since the motion energy detector is formally equivalentto an elaborated Reichardt motion detector [9] the structureof the motion energy detector with divisive normalization isalso similar to the phase-based motion detector
4 Exploratory Results
In this section we apply the phase-based motion detectionalgorithm on several video sequences and demonstrate itsefficiency and effectiveness in detecting local motion Themotion detection algorithm is compared to two well-knownbiological motion detectors namely the Reichardt motiondetector and the Barlow-Levick motion detector We thenshow that the detected local motion can be used in motionsegmentation tasks The effectiveness of the segmentationis compared to segmentation using motion informationobtained from a widely used optic flow based algorithmavailable in the literature [4]
Computational Intelligence and Neuroscience 11
(f) (e)
(b)
(a)
(d)
(c)
minus120587 minus1205872 0 1205872
10080604020
y (pi
xel)
100 15050x (pixel)
0020406081
15
10
5
l
01234567+
15
10
5
l
0510152025+
minus6
minus4
minus2
0
2
4
6
181716151413121110
9876543210
l
2 4 6 8 10 12 14 16 18 20 22 240k
minus6minus4minus20246
minus6minus4minus20246
120596x
minus120587 minus1205872 0 1205872
120596x
1205872
0
minus1205872
minus120587
120596y
1205872
0
minus1205872
minus120587
120596y
10 15 20 255k
10 20 255 15k
Figure 6 Evaluating the motion detection algorithm at time 1199050(see also Supplementary Video S1) (a) A still from a 156 times 112-pixel video at
time 1199050 (b) (119889120601
119896119897119889119905)(120596
119909 120596119910 1199050) for all 26times19 blocks each of size 32times32The blocks are concatenated to create a large ldquophase imagerdquo with their
relative neighbors kept (c) (1198891206011211
119889119905)(120596119909 120596119910 1199050) shown in the (120596
119909 120596119910) isin D2 space (d) (119889120601
236119889119905)(120596
119909 120596119910 1199050) shown in the (120596
119909 120596119910) isin D2
space (e) Phase Motion Indicator (PMI) for all blocks at 1199050 Each pixel (119896 119897) is color coded for PMI
119896119897 (f) max
(120596119909 120596119910)isinD2 |(119889120601119896119897119889119905)(120596
119909 120596119910 1199050)|
for all blocks at 1199050 Each pixel (119896 119897) is color coded for this maximum value of block (119896 119897)
g1(t)g2(t) g1(t) g2(t)
times times
minus++
SpatialGaborfilter
SpatialGaborfilter
r1(t) r2(t)
u(x y t)
Figure 7 Diagram of an elaborated Reichardt motion detector Itconsists of a quadrature pair of Gabor filters whose outputs 119903
1and 1199032
provide inputs to the high-pass filters 1198921(119905) and the low-pass filters
1198922(119905)
Table 1 Processing Speeds of the proposed algorithm the Reichardtmotion detector and the Barlow-Levick Motion detector
Screen size Processing capability (frames per second)Proposed Reichardt Barlow-Levick
320 times 240 420 790 800720 times 576 135 685 6901280 times 720 60 274 2931920 times 1080 27 191 194
41 Efficient Parallel Implementation The algorithm wasimplemented in PyCUDA [23] and tested on an NVIDIAGeForce GTX TITAN GPU All computations use singleprecision floating points The processing speeds of the algo-rithm for several screen sizes are listed in Table 1 Clearlythe proposed phase-based motion detection algorithm hasreal-time capability to process video even with full HighDefinition screen size
For comparison we implemented the Reichardt motiondetector and the Barlow-Levick motion detector Theirrespective diagrams are shown in Figure 8 Note that we
12 Computational Intelligence and Neuroscience
ukl(t) uk+1l(t)
g1(t)
g2(t)
g1(t)
g2(t)
+ minus+
timestimes
(a)
minus
minus minus
+
ukl(t) uk+1l(t)
g1(t)g1(t)
g2(t) g2(t)
++
+
(b)
Figure 8 (a) A Reichardt motion detector detecting motion between the two points 119906119896119897and 119906
119896+1119897in the blurred and downsampled video
(b) A Barlow-Levick motion detector detecting motion between the two points 119906119896119897and 119906
119896+1119897in the blurred and downsampled video 119892
1(119905)
denotes a high-pass filter and 1198922(119905) denotes a low-pass filter
moved the high-pass filter ℎ1(119905) to the front of the low-pass fil-
ter ℎ2(119905) This configuration provides a superior performance
to the one in Figure 7 For both the Reichardt and the Barlow-Levick detectors the videos are first blurred by a Gaussianfilter (with the same variance as the Gaussian window in (30)used for the phase-based motion detector) and subsampledat the center of each overlapping block in the phase-basedmotion detector The subsampled video then provides inputsto two 2D arrays of the circuits shown in Figure 8 one forthe horizontal direction and one for the vertical directionThe outputs of the horizontal and vertical motion circuitsform an array of motion vectors that indicate the strengthand direction of motion The three tested motion detectorshave the same number of outputs as a result The processingspeeds of the Reichardt motion detector and the Barlow-Levick motion detector both implemented in PyCUDA andtested on the same GPU are shown in Table 1
Note that the Reichardt motion detector and the Barlow-Levick motion detector are highly efficient due to the sim-plicity of their algorithmsThe phase-basedmotion detectionalgorithm however is a much more sophisticated algorithmand yet it can be implemented in real-time using parallelcomputing devices
The fast GPU implementation is based on the FFTand Matrix-Matrix multiplication It is expected that thoseoperations can be efficiently implemented in hardware forexample FPGA
42 Examples of Phased-BasedMotionDetection Weappliedour motion detection algorithm on video sequences of theChange Detection 2014 dataset [22] that did not exhibitcamera egomotion For these video sequences the standarddeviation of the Gaussian window functions was set to 4
pixels and the block size was chosen to be 32 times 32 pixelsThreshold and normalization parameters were kept the samewith the exception of the ldquothermal videordquo (in order to dealwith larger background noise levels see below) We alsotested the same video sequences using the Reichardt motiondetector and the Barlow-Levick motion detector For theReichardt detector the high-pass filters were chosen as firstorder filters with a time constant of 200 milliseconds and thelow-pass filters were chosen as first order filters with a timeconstant of 300 milliseconds (assuming that the frame rate is50 frames per second)Thresholdwas set to 2 For the Barlow-Levick motion detector the time constant of the first high-pass filters was set to 250 milliseconds The low-pass filterswere the same as in the Reichardt detectorThreshold was setto 2
The first video was taken from a highway surveillancecamera (ldquohighway videordquo) under good illumination con-ditions and high contrast The video had moderate noiseparticularly on the road surface The detected motion isshown in the top left panel of Figure 9 (see SupplementaryVideo S2 for full video) The phase-based motion detectionalgorithm captured both the moving cars and the tree leavesmoving (due to the wind) In the time interval between the9th and 10th second the camera was slightly moved left-and right-wards within 5 frames again possibly due to thewind Movement due to this shift was captured by themotiondetection algorithm and the algorithm was fast enough tocorrectly determine the direction of this movement In thisvideo we already noted that this algorithm suffers from theaperture problem For example in front of the van where along horizontal edge is present the detectedmotion ismostlypointing downwards In addition to moving downwards theedge is alsomoving to the left howeverThis is expected since
Computational Intelligence and Neuroscience 13
(a) (b) (c)
Figure 9 Top motion detection of the ldquohighway videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) Red arrows indicate detected motion and its direction Bottom the contrast of the video was artificiallyreduced by 5-fold and the mean was reduced to 35 of the original The red arrows are duplicated from the motion detection result on theoriginal video as in top Blue arrows are the result of motion detection on the video with reduced contrast If motion is detected both in theoriginal video and in the video with reduced contrast then the arrow is shown in magenta (as a mix of blue and red) (see SupplementaryVideo S2 for full video)
the algorithm only detects motion locally and does not takeinto account the overall shape of any object
For comparison the motion detection results for theReichardt motion detector and the Barlow-Levick motiondetector are shown in the top middle and top right panelof Figure 9 (see Supplementary Video S2 for full video) TheReichardt detector performed relatively well when vehiclesare moving faster but the direction it predicts for vehiclesmoving slower for example on the back of the image is notaccurate In addition motion is still detected in some partsof the screen where the vehicles have just passed by Thedetection result for the Barlow-Levick motion detector waspoorer In particular the response to OFF edge movement isalways the opposite to the actual movement direction
We then squeezed the range of the screen intensity from[0 1] to [02 04] resulting in a video with a lower meanluminance and lower contrast The motion detection resultson the low-contrast video are shown in the bottom 3 panelsof Figure 9 (see Supplementary Video S2 for full video) Forreference the motion detection results for the original videoare shown in red arrows Motion detected in the low-contrastvideo is shown in blue arrows if no motion is detected forthe block in the original video If motion is detected in boththe original and the low-contrast video the arrows are shownin magenta Figure 9 clearly shows that while the motiondetection performance is degraded for all three detectors thephase-basedmotion detector performed still quite well in thelow-contrast video detecting most of the moving vehicles
The other two detectors missed many of the blocks wheremotion was detected in the original movie
To quantify how well each detector works under differentcontrast conditions we computed the ratio of unthresholdedoutput values of eachmotion detector between lower contrastand full contrast video For example in the case of phase-basedmotion detector we computed for each block the ratiobetween the PMI index for the lower contrast video and thatfor the full contrast videoThe ratios are then averaged acrossall blocks where motion is detected in the full contrast videoThis average for different contrasts is shown in Figure 10 bythe blue curve In the ideal case when the normalizationconstant 120598 is 0 the phase detector should produce invariantresponses to videos with different contrast The curve shownhere is mainly due to a nonzero 120598The ratios for the Reichardtmotion detector and the Barlow-Levick motion detectorare computed similarly and are shown in red and yellowrespectively It is clear that the phase-based motion detectorhas a superior performance across a range of contrast valuesAt 20 contrast the phase-based detector still has 50 ofthe PMI index value for full contrast video As expectedthe response of Barlow-Levick motion detector is linear withrespect to contrast and the Reichardt motion detector has aquadratic relation to contrast For a fixed threshold value alarger ratio equates to amore consistent performance at lowercontrast
The second video was captured by a surveillance camerain a train station (ldquotrain station videordquo) The video was
14 Computational Intelligence and Neuroscience
Phase-basedReichardtBarlow-Levick
0
02
04
06
08
1
Ratio
40 60 80200 100Contrast ()
Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined
under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm
The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)
The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression
With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)
43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient
objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area
For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold
We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually
We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm
We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)
These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm
5 Discussion
Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes
Computational Intelligence and Neuroscience 15
(a) (b) (c)
Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)
(a) (b) (c)
Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)
by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to
models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]
We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in
16 Computational Intelligence and Neuroscience
(a) (b) (c)
Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)
Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)
phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in
Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)
modeling in vivo biological vision systems This is consistentwith other recent findings [15]
Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure
Computational Intelligence and Neuroscience 17
(a) (b)
Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)
(a) (b)
Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)
(a) (b)
Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)
Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase
itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third
18 Computational Intelligence and Neuroscience
(a) (b)
Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)
instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection
The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions
Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors
We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well
We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It
can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes
Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems
Appendix
The Derivative of the Local Phase
Let
119886 (120596119909 120596119910 119905)
= int
R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596
119909119909 + 120596119910119910) 119889119909 119889119910
119887 (120596119909 120596119910 119905)
= minusint
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
(A1)
and therefore
11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)
= 119886 (120596119909 120596119910 119905) + 119895119887 (120596
119909 120596119910 119905)
(A2)
The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596
119909
120596119910 119905) 119886(120596
119909 120596119910 119905)) and therefore
11988912060100
119889119905
=
119886 (119889119887119889119905) minus 119887 (119889119886119889119905)
[119886]2+ [119887]2
(A3)
Computational Intelligence and Neuroscience 19
Now
119889119886
119889119905
(120596119909 120596119910 119905)
= int
R2
120597119906
120597119905
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
(A4)
For a purely translational signal with instantaneous spatialshift given by (119904
119909(119905) 119904119910(119905)) within the support of the window
we have
119889119886
119889119905
(120596119909 120596119910 119905)
= minusint
R2
119889119904119909
119889119905
(119905)
120597119906
120597119909
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus int
R2
119889119904119910
119889119905
(119905)
120597119906
120597119910
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119909
119889119904119909
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119910
119889119904119910
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199033=1199033(120596119909120596119910119905)
+ 120596119909
119889119904119909
119889119905
119887
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199034=1199034(120596119909120596119910119905)
+ 120596119910
119889119904119910
119889119905
119887
(A5)
Similarly
119889119887
119889119905
(120596119909 120596119910 119905) = minus
119889119904119909
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199035=1199035(120596119909120596119910119905)
minus 120596119909
119889119904119909
119889119905
119886 minus
119889119904119910
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199036=1199036(120596119909120596119910119905)
minus 120596119910
119889119904119910
119889119905
119886
(A6)
Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to
11988912060100
119889119905
(120596119909 120596119910 119905) =
minus (119889119904119909119889119905) [119903
3119887 + 1199035119886 + 120596119909 [
119886]2+ 120596119909 [
119887]2] minus (119889119904
119910119889119905) [119903
4119887 + 1199036119886 + 120596119910 [
119886]2+ 120596119910 [
119887]2]
[119886]2+ [119887]2
= minus
119889119904119909
119889119905
120596119909minus
119889119904119910
119889119905
120596119910minus
119889119904119909
119889119905
sdot
1199033119887 + 1199035119886
[119886]2+ [119887]2minus
119889119904119910
119889119905
sdot
1199034119887 + 1199036119886
[119886]2+ [119887]2
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
v00(120596119909120596119910119905)
(A7)
Disclosure
The authorrsquos names are alphabetically listed
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgment
The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232
References
[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010
[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010
[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014
[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010
20 Computational Intelligence and Neuroscience
[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013
[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014
[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956
[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010
[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985
[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965
[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981
[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015
[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014
[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999
[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015
[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001
[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996
[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980
[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013
[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013
[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014
[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014
[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach
to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012
[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990
[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987
[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998
Submit your manuscripts athttpwwwhindawicom
Computer Games Technology
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Distributed Sensor Networks
International Journal of
Advances in
FuzzySystems
Hindawi Publishing Corporationhttpwwwhindawicom
Volume 2014
International Journal of
ReconfigurableComputing
Hindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Applied Computational Intelligence and Soft Computing
thinspAdvancesthinspinthinsp
Artificial Intelligence
HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014
Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Journal of
Computer Networks and Communications
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation
httpwwwhindawicom Volume 2014
Advances in
Multimedia
International Journal of
Biomedical Imaging
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
ArtificialNeural Systems
Advances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience
Industrial EngineeringJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Human-ComputerInteraction
Advances in
Computer EngineeringAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
8 Computational Intelligence and Neuroscience
The Radon transform of the change of phase in thedomain 119862 is given by
(R119889120601119896119897
119889119905
) (120588 120579 119905) = int
R
119889120601119896119897
119889119905
(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588
sdot sin 120579 + 119904 sdot cos 120579 119905) 1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579
+ 119904 sdot cos 120579) 119889119904
(34)
where
1119862(120596119909 120596119910) =
1 if (120596119909 120596119910) isin 119862
0 otherwise(35)
The Radon transform (R(119889120601119896119897119889119905))(120588 120579 119905) evaluated at
a particular point (1205880 1205790 1199050) is essentially an integral
of (119889120601119896119897119889119905)(120596
119909 120596119910 1199050) along a line oriented at angle
1205872 + 1205790with the 120596
119909axis and at distance |120588
0| along the
(cos(1205790) sin(120579
0)) direction from (0 0)
If for a particular 119896 and 119897 we have (119889120601119896119897119889119905)(120596
119909 120596119910 119905) =
minusV119909(119905)120596119909minus V119910(119905)120596119910 we have
(R (119889120601119896119897119889119905)) (120588 120579 119905)
c (120588 120579)
= 120588 [minusV119909(119905) cos 120579 minus V
119910(119905) sin 120579]
(36)
where
c (120588 120579)
= int
R
1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579 + 119904 sdot cos 120579) 119889119904
(37)
c(120588 120579) is a correction term due to different length of lineintegrals for different values of (120588 120579) in the bounded domain119862
After computing the Radon transform of (119889120601119896119897119889119905)(120596
119909
120596119910 119905) for every block (119896 119897) at time 119905
0 we compute the Phase
Motion Indicator (PMI) defined as
PMI119896119897= max120579isin[0120587)
sum
120588
100381610038161003816100381610038161003816100381610038161003816
(R (119889120601119896119897119889119905)) (120588 120579 119905
0)
c (120588 120579)
100381610038161003816100381610038161003816100381610038161003816
(38)
If the PMI119896119897
is larger than a chosen threshold motion isdeemed to occur in block (119896 119897) at time 119905
0
Using the Radon transform makes it easier to separaterigid motion from noise Since the phase is quite sensitiveto noise particularly when the amplitude is very smallthe change of phase under noise may have comparablemagnitude to that due to motion as mentioned earlier Thechange of phase under noise however does not possess thestructure suggested by (29) in the (120596
119909 120596119910) domain Instead
it appears to be more randomly distributed Consequently
the PMI value is comparatively small for these blocks (see alsoSection 333)
Moreover the direction of motion for block (119896 119897) wheremotion is detected can be easily computed as
120579119896119897
= 120587(
sign (sum120588gt0
(R (119889120601119896119897119889119905)) (120588 120572
119896119897 1199050) c (120588 120572
119896119897)) + 1
2
)
+ 120572119896119897
(39)
where
120572119896119897= argmax120579isin[0120587)
sum
120588
100381610038161003816100381610038161003816100381610038161003816
(R (119889120601119896119897119889119905)) (120588 120579 119905
0)
c (120588 120579)
100381610038161003816100381610038161003816100381610038161003816
(40)
This follows from (36)
332 The Phase-Based Motion Detection Algorithm We for-mally summarize the above analysis as Algorithm 1 Figure 5shows a schematic diagram of the proposed phase-basedmotion detection algorithm
The algorithm is subdivided into two parts The first partcomputes local phase changes and the second part is thephase-based motion detector
In the first part the screen is divided into overlappingblocks For example the red green and blue blocks in theplane ldquodivide into overlapping blocksrdquo correspond to thesquares of the same color covering the video stream AGaussian window is then applied on each block followed bya 2D FFT operation that is used to extract the local phase Atemporal high-pass filter is then employed to extract phasechanges
In the second part the PMI is evaluated for each blockbased on the Radon transform of the local phase changes ineach block Motion is detected for blocks with PMI largerthan a preset threshold and the direction of motion iscomputed as in (39)
It is easy to notice that the algorithm can be highlyparallelized
333 Example We provide an illustrative example inFigure 6 showing how motion is detected using Algorithm 1The full video of this example can be found in SupplementaryVideo S1 see Supplementary Material available online athttpdxdoiorg10115520167915245 Figure 6(a) depicts astill from the ldquohighway videordquo in the Change Detection 2014dataset [22] evaluated at a particular time 119905
0 As suggested
by the algorithm the screen in Figure 6(a) is divided into26 times 19 overlapping blocks and the window functions areapplied to each block Local phases can then be extractedfrom the 2DFFT of eachwindowed block and the local phasechanges are obtained by temporal high-pass filtering Thephase change is shown in Figure 6(b) for all blocks with block(12 11) enlarged in Figure 6(c) and block (23 6) enlarged inFigure 6(d) (see also the plane ldquo2D FFT and extract phasechangerdquo in Figure 5) Note that at the time of the videoframe block (12 11) covers a part of the vehicle in motion
Computational Intelligence and Neuroscience 9
Input Visual stream or Video data 119906(119909 119910 119905)Output Detected MotionConstruct Gaussian window of size119872times119872 denote as 119908for 119905 = Δ119879 2Δ119879 119873Δ119879 do
Divide the screen of frame at time 119905 into multiple119872times119872 blocks 119906119896119897(119905) with overlaps
foreach block 119906119896119897(119905) do
Multiply pixel-wise the block 119906119896119897(119905) with the Gaussian window 119906
119896119897(119905)119908
Take119872times119872 2D FFT of 119906119896119897(119905)119908 denote phase as 120601
119896119897(120596119909 120596119910 119905) and amplitude as 119860
119896119897(120596119909 120596119910 119905)
foreach frequency (120596119909 120596119910) do
Highpass filter temporally 120601119896119897(120596119909 120596119910 119905) denote 1206011015840
119896119897(120596119909 120596119910 119905)
Optionally denoise by 1206011015840119896119897(120596119909 120596119910 119905) larr 120601
1015840
119896119897(120596119909 120596119910 119905)(119860
119896119897(120596119909 120596119910 119905)((1119872
2) sum(120596119909 120596119910)isinD
2 119860119896119897(120596119909 120596119910 119905) + 120598))
endCompute the radon transform (R120601
1015840
119896119897)(120588 120579 119905)
Compute PMI119896119897(119905) according to (38)
if PMI119896119897(119905) gt threshold then
Motion Detected at block (119896 119897) at time 119905Compute direction of motion according to (39)
endend
end
Algorithm 1 Phase-based motion detection algorithm using the FFT
y
120596x
120596y
y
t
120596x
120596y
x
y
x
y
x
x
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
Divided intooverlapping blocks
Apply window function
2D FFT andextract phase change
Radon transformand extract strength
and orientation of plane
Thresholding
Computephase
change
The phase-baseddetector
Figure 5 Schematic diagram of proposed phase-based motion detection algorithm
10 Computational Intelligence and Neuroscience
in the front and block (23 6) corresponds to an area of thehighway pavement where no motion occurs
Figure 6(f) depicts for each block (119896 119897) the maximumphase change over all (120596
119909 120596119910) isin D2 that is
max(120596119909120596119910)isinD2
10038161003816100381610038161003816100381610038161003816
119889120601119896119897
119889119905
(120596119909 120596119910 1199050)
10038161003816100381610038161003816100381610038161003816
(41)
We observe from the figure that for regions with lowamplitude such as the region depicting the road whenthe normalization constant is absent the derivative of thephase can be noisy For these blocks the maximum of|(119889120601119896119897119889119905)(120596
119909 120596119910 1199050)| over all (120596
119909 120596119910) isin D2 is comparable
to the maximum obtained for blocks that cover the vehiclesin motion
However (29) suggests that the local phase change frommultiple filter pairs centered at the same spatial position(119896 119897) can provide a constraint to robustly estimate motionand its direction Given the block structure employed in thecomputation of the local phase it is natural to utilize phasechange information from multiple sources
Indeed if for a particular block (119896 119897) (119889120601119896119897119889119905)(120596
119909
120596119910 119905) = minusV
119909(119905)120596119909minus V119910(119905)120596119910 then it is easy to see that (119889120601
119896119897
119889119905)(120596119909 120596119910 119905) will be zero on the line V
119909(119905)120596119909+ V119910(119905)120596119910= 0
and have opposite sign on either side of this line For examplein Figures 6(b) and 6(c) (119889120601
119896119897119889119905)(120596
119909 120596119910 1199050) clearly exhibits
this property for blocks that cover a vehicle in motion ThePMI is a tool to evaluate this property
Finally the PMIs for all blocks are shown compactlyin a heat map in Figure 6(e) The figure shows clearly thatthe blocks corresponding to the two moving vehicles have ahigh PMI value while the stationary background areas havea low PMI value allowing one to easily detect motion byemploying simple thresholding (see also the plane ldquoRadontransform and extract strength orientation of planerdquo inFigure 5) In addition the orientation ofmotion in each blockis readily observable even by inspection in Figure 6(b) by aline separating the yellow part and blue part in each blockFurther results about the direction of motion are presentedin Section 4
34 Relationship to Biological Motion Detectors A straight-forward way to implement local motion detectors is to applya complex-valued Gabor receptive field (5) to the video signal119906 and then take the derivative of the phase with respect totime or apply a high-pass filter on the phase to approximatethe derivative
We present here an alternate implementation withoutexplicitly computing the phaseThiswill elucidate the relationbetween the phase-based motion detector presented in theprevious section and some elementary motion detectionmodels used in biology such as theReichardtmotion detector[7] and motion energy detector [9]
Assuming that the local phase 12060100(120596119909 120596119910 119905) is differen-
tiable we have (see also the Appendix)
11988912060100(120596119909 120596119910 119905)
119889119905
=
(119889119887 (120596119909 120596119910 119905) 119889119905) 119886 (120596
119909 120596119910 119905) minus (119889119886 (120596
119909 120596119910 119905) 119889119905) 119887 (120596
119909 120596119910 119905)
[119886 (120596119909 120596119910 119905)]
2
+ [119887 (120596119909 120596119910 119905)]
2 (42)
where 119886(120596119909 120596119910 119905) and 119887(120596
119909 120596119910 119905) are respectively the real
and imaginary parts of 11986000(120596119909 120596119910 119905)11989011989512060100(120596119909120596119910119905)
We notice that the denominator of (42) is the square ofthe local amplitude of 119906 and the numerator is of the formof a second order Volterra kernel This suggests that thetime derivative of the local phase can be viewed as a secondorder Volterra kernel that processes two normalized spatiallyfiltered inputs V
1and V2
We consider an elaborated Reichardt motion detector asshown in Figure 7 It is equipped with a quadrature pair ofGabor filters whose outputs are 119903
1(119905) = 119886(120596
119909 120596119910 119905) and
1199032(119905) = 119887(120596
119909 120596119910 119905) respectively for a particular value of
(120596119909 120596119910) The pair of Gabor filters that provide these outputs
are the real and imaginary parts of119908(119909 119910)119890minus119895(120596119909119909+120596119910119910) It also
consists of a temporal high-pass 1198921(119905) and temporal low-
pass filter 1198922(119905) [9] The output of the elaborated Reichardt
detector follows the diagram in Figure 7 and can be expressedas
(1199032lowast 1198921) (119905) (119903
1lowast 1198922) (119905) minus (119903
1lowast 1198921) (119905) (119903
2lowast 1198922) (119905) (43)
The response can also be characterized by a second orderVolterra kernel We notice the striking similarity between
(43) and the numerator of (42) In fact the phase-basedmotion detector shares some properties with the Reichardtmotion detector For example it is straightforward to see thata single phase-basedmotion detector is tuned to the temporalfrequency of a moving sinusoidal grating
Since the motion energy detector is formally equivalentto an elaborated Reichardt motion detector [9] the structureof the motion energy detector with divisive normalization isalso similar to the phase-based motion detector
4 Exploratory Results
In this section we apply the phase-based motion detectionalgorithm on several video sequences and demonstrate itsefficiency and effectiveness in detecting local motion Themotion detection algorithm is compared to two well-knownbiological motion detectors namely the Reichardt motiondetector and the Barlow-Levick motion detector We thenshow that the detected local motion can be used in motionsegmentation tasks The effectiveness of the segmentationis compared to segmentation using motion informationobtained from a widely used optic flow based algorithmavailable in the literature [4]
Computational Intelligence and Neuroscience 11
(f) (e)
(b)
(a)
(d)
(c)
minus120587 minus1205872 0 1205872
10080604020
y (pi
xel)
100 15050x (pixel)
0020406081
15
10
5
l
01234567+
15
10
5
l
0510152025+
minus6
minus4
minus2
0
2
4
6
181716151413121110
9876543210
l
2 4 6 8 10 12 14 16 18 20 22 240k
minus6minus4minus20246
minus6minus4minus20246
120596x
minus120587 minus1205872 0 1205872
120596x
1205872
0
minus1205872
minus120587
120596y
1205872
0
minus1205872
minus120587
120596y
10 15 20 255k
10 20 255 15k
Figure 6 Evaluating the motion detection algorithm at time 1199050(see also Supplementary Video S1) (a) A still from a 156 times 112-pixel video at
time 1199050 (b) (119889120601
119896119897119889119905)(120596
119909 120596119910 1199050) for all 26times19 blocks each of size 32times32The blocks are concatenated to create a large ldquophase imagerdquo with their
relative neighbors kept (c) (1198891206011211
119889119905)(120596119909 120596119910 1199050) shown in the (120596
119909 120596119910) isin D2 space (d) (119889120601
236119889119905)(120596
119909 120596119910 1199050) shown in the (120596
119909 120596119910) isin D2
space (e) Phase Motion Indicator (PMI) for all blocks at 1199050 Each pixel (119896 119897) is color coded for PMI
119896119897 (f) max
(120596119909 120596119910)isinD2 |(119889120601119896119897119889119905)(120596
119909 120596119910 1199050)|
for all blocks at 1199050 Each pixel (119896 119897) is color coded for this maximum value of block (119896 119897)
g1(t)g2(t) g1(t) g2(t)
times times
minus++
SpatialGaborfilter
SpatialGaborfilter
r1(t) r2(t)
u(x y t)
Figure 7 Diagram of an elaborated Reichardt motion detector Itconsists of a quadrature pair of Gabor filters whose outputs 119903
1and 1199032
provide inputs to the high-pass filters 1198921(119905) and the low-pass filters
1198922(119905)
Table 1 Processing Speeds of the proposed algorithm the Reichardtmotion detector and the Barlow-Levick Motion detector
Screen size Processing capability (frames per second)Proposed Reichardt Barlow-Levick
320 times 240 420 790 800720 times 576 135 685 6901280 times 720 60 274 2931920 times 1080 27 191 194
41 Efficient Parallel Implementation The algorithm wasimplemented in PyCUDA [23] and tested on an NVIDIAGeForce GTX TITAN GPU All computations use singleprecision floating points The processing speeds of the algo-rithm for several screen sizes are listed in Table 1 Clearlythe proposed phase-based motion detection algorithm hasreal-time capability to process video even with full HighDefinition screen size
For comparison we implemented the Reichardt motiondetector and the Barlow-Levick motion detector Theirrespective diagrams are shown in Figure 8 Note that we
12 Computational Intelligence and Neuroscience
ukl(t) uk+1l(t)
g1(t)
g2(t)
g1(t)
g2(t)
+ minus+
timestimes
(a)
minus
minus minus
+
ukl(t) uk+1l(t)
g1(t)g1(t)
g2(t) g2(t)
++
+
(b)
Figure 8 (a) A Reichardt motion detector detecting motion between the two points 119906119896119897and 119906
119896+1119897in the blurred and downsampled video
(b) A Barlow-Levick motion detector detecting motion between the two points 119906119896119897and 119906
119896+1119897in the blurred and downsampled video 119892
1(119905)
denotes a high-pass filter and 1198922(119905) denotes a low-pass filter
moved the high-pass filter ℎ1(119905) to the front of the low-pass fil-
ter ℎ2(119905) This configuration provides a superior performance
to the one in Figure 7 For both the Reichardt and the Barlow-Levick detectors the videos are first blurred by a Gaussianfilter (with the same variance as the Gaussian window in (30)used for the phase-based motion detector) and subsampledat the center of each overlapping block in the phase-basedmotion detector The subsampled video then provides inputsto two 2D arrays of the circuits shown in Figure 8 one forthe horizontal direction and one for the vertical directionThe outputs of the horizontal and vertical motion circuitsform an array of motion vectors that indicate the strengthand direction of motion The three tested motion detectorshave the same number of outputs as a result The processingspeeds of the Reichardt motion detector and the Barlow-Levick motion detector both implemented in PyCUDA andtested on the same GPU are shown in Table 1
Note that the Reichardt motion detector and the Barlow-Levick motion detector are highly efficient due to the sim-plicity of their algorithmsThe phase-basedmotion detectionalgorithm however is a much more sophisticated algorithmand yet it can be implemented in real-time using parallelcomputing devices
The fast GPU implementation is based on the FFTand Matrix-Matrix multiplication It is expected that thoseoperations can be efficiently implemented in hardware forexample FPGA
42 Examples of Phased-BasedMotionDetection Weappliedour motion detection algorithm on video sequences of theChange Detection 2014 dataset [22] that did not exhibitcamera egomotion For these video sequences the standarddeviation of the Gaussian window functions was set to 4
pixels and the block size was chosen to be 32 times 32 pixelsThreshold and normalization parameters were kept the samewith the exception of the ldquothermal videordquo (in order to dealwith larger background noise levels see below) We alsotested the same video sequences using the Reichardt motiondetector and the Barlow-Levick motion detector For theReichardt detector the high-pass filters were chosen as firstorder filters with a time constant of 200 milliseconds and thelow-pass filters were chosen as first order filters with a timeconstant of 300 milliseconds (assuming that the frame rate is50 frames per second)Thresholdwas set to 2 For the Barlow-Levick motion detector the time constant of the first high-pass filters was set to 250 milliseconds The low-pass filterswere the same as in the Reichardt detectorThreshold was setto 2
The first video was taken from a highway surveillancecamera (ldquohighway videordquo) under good illumination con-ditions and high contrast The video had moderate noiseparticularly on the road surface The detected motion isshown in the top left panel of Figure 9 (see SupplementaryVideo S2 for full video) The phase-based motion detectionalgorithm captured both the moving cars and the tree leavesmoving (due to the wind) In the time interval between the9th and 10th second the camera was slightly moved left-and right-wards within 5 frames again possibly due to thewind Movement due to this shift was captured by themotiondetection algorithm and the algorithm was fast enough tocorrectly determine the direction of this movement In thisvideo we already noted that this algorithm suffers from theaperture problem For example in front of the van where along horizontal edge is present the detectedmotion ismostlypointing downwards In addition to moving downwards theedge is alsomoving to the left howeverThis is expected since
Computational Intelligence and Neuroscience 13
(a) (b) (c)
Figure 9 Top motion detection of the ldquohighway videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) Red arrows indicate detected motion and its direction Bottom the contrast of the video was artificiallyreduced by 5-fold and the mean was reduced to 35 of the original The red arrows are duplicated from the motion detection result on theoriginal video as in top Blue arrows are the result of motion detection on the video with reduced contrast If motion is detected both in theoriginal video and in the video with reduced contrast then the arrow is shown in magenta (as a mix of blue and red) (see SupplementaryVideo S2 for full video)
the algorithm only detects motion locally and does not takeinto account the overall shape of any object
For comparison the motion detection results for theReichardt motion detector and the Barlow-Levick motiondetector are shown in the top middle and top right panelof Figure 9 (see Supplementary Video S2 for full video) TheReichardt detector performed relatively well when vehiclesare moving faster but the direction it predicts for vehiclesmoving slower for example on the back of the image is notaccurate In addition motion is still detected in some partsof the screen where the vehicles have just passed by Thedetection result for the Barlow-Levick motion detector waspoorer In particular the response to OFF edge movement isalways the opposite to the actual movement direction
We then squeezed the range of the screen intensity from[0 1] to [02 04] resulting in a video with a lower meanluminance and lower contrast The motion detection resultson the low-contrast video are shown in the bottom 3 panelsof Figure 9 (see Supplementary Video S2 for full video) Forreference the motion detection results for the original videoare shown in red arrows Motion detected in the low-contrastvideo is shown in blue arrows if no motion is detected forthe block in the original video If motion is detected in boththe original and the low-contrast video the arrows are shownin magenta Figure 9 clearly shows that while the motiondetection performance is degraded for all three detectors thephase-basedmotion detector performed still quite well in thelow-contrast video detecting most of the moving vehicles
The other two detectors missed many of the blocks wheremotion was detected in the original movie
To quantify how well each detector works under differentcontrast conditions we computed the ratio of unthresholdedoutput values of eachmotion detector between lower contrastand full contrast video For example in the case of phase-basedmotion detector we computed for each block the ratiobetween the PMI index for the lower contrast video and thatfor the full contrast videoThe ratios are then averaged acrossall blocks where motion is detected in the full contrast videoThis average for different contrasts is shown in Figure 10 bythe blue curve In the ideal case when the normalizationconstant 120598 is 0 the phase detector should produce invariantresponses to videos with different contrast The curve shownhere is mainly due to a nonzero 120598The ratios for the Reichardtmotion detector and the Barlow-Levick motion detectorare computed similarly and are shown in red and yellowrespectively It is clear that the phase-based motion detectorhas a superior performance across a range of contrast valuesAt 20 contrast the phase-based detector still has 50 ofthe PMI index value for full contrast video As expectedthe response of Barlow-Levick motion detector is linear withrespect to contrast and the Reichardt motion detector has aquadratic relation to contrast For a fixed threshold value alarger ratio equates to amore consistent performance at lowercontrast
The second video was captured by a surveillance camerain a train station (ldquotrain station videordquo) The video was
14 Computational Intelligence and Neuroscience
Phase-basedReichardtBarlow-Levick
0
02
04
06
08
1
Ratio
40 60 80200 100Contrast ()
Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined
under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm
The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)
The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression
With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)
43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient
objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area
For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold
We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually
We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm
We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)
These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm
5 Discussion
Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes
Computational Intelligence and Neuroscience 15
(a) (b) (c)
Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)
(a) (b) (c)
Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)
by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to
models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]
We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in
16 Computational Intelligence and Neuroscience
(a) (b) (c)
Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)
Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)
phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in
Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)
modeling in vivo biological vision systems This is consistentwith other recent findings [15]
Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure
Computational Intelligence and Neuroscience 17
(a) (b)
Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)
(a) (b)
Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)
(a) (b)
Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)
Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase
itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third
18 Computational Intelligence and Neuroscience
(a) (b)
Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)
instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection
The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions
Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors
We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well
We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It
can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes
Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems
Appendix
The Derivative of the Local Phase
Let
119886 (120596119909 120596119910 119905)
= int
R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596
119909119909 + 120596119910119910) 119889119909 119889119910
119887 (120596119909 120596119910 119905)
= minusint
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
(A1)
and therefore
11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)
= 119886 (120596119909 120596119910 119905) + 119895119887 (120596
119909 120596119910 119905)
(A2)
The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596
119909
120596119910 119905) 119886(120596
119909 120596119910 119905)) and therefore
11988912060100
119889119905
=
119886 (119889119887119889119905) minus 119887 (119889119886119889119905)
[119886]2+ [119887]2
(A3)
Computational Intelligence and Neuroscience 19
Now
119889119886
119889119905
(120596119909 120596119910 119905)
= int
R2
120597119906
120597119905
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
(A4)
For a purely translational signal with instantaneous spatialshift given by (119904
119909(119905) 119904119910(119905)) within the support of the window
we have
119889119886
119889119905
(120596119909 120596119910 119905)
= minusint
R2
119889119904119909
119889119905
(119905)
120597119906
120597119909
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus int
R2
119889119904119910
119889119905
(119905)
120597119906
120597119910
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119909
119889119904119909
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119910
119889119904119910
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199033=1199033(120596119909120596119910119905)
+ 120596119909
119889119904119909
119889119905
119887
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199034=1199034(120596119909120596119910119905)
+ 120596119910
119889119904119910
119889119905
119887
(A5)
Similarly
119889119887
119889119905
(120596119909 120596119910 119905) = minus
119889119904119909
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199035=1199035(120596119909120596119910119905)
minus 120596119909
119889119904119909
119889119905
119886 minus
119889119904119910
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199036=1199036(120596119909120596119910119905)
minus 120596119910
119889119904119910
119889119905
119886
(A6)
Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to
11988912060100
119889119905
(120596119909 120596119910 119905) =
minus (119889119904119909119889119905) [119903
3119887 + 1199035119886 + 120596119909 [
119886]2+ 120596119909 [
119887]2] minus (119889119904
119910119889119905) [119903
4119887 + 1199036119886 + 120596119910 [
119886]2+ 120596119910 [
119887]2]
[119886]2+ [119887]2
= minus
119889119904119909
119889119905
120596119909minus
119889119904119910
119889119905
120596119910minus
119889119904119909
119889119905
sdot
1199033119887 + 1199035119886
[119886]2+ [119887]2minus
119889119904119910
119889119905
sdot
1199034119887 + 1199036119886
[119886]2+ [119887]2
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
v00(120596119909120596119910119905)
(A7)
Disclosure
The authorrsquos names are alphabetically listed
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgment
The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232
References
[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010
[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010
[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014
[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010
20 Computational Intelligence and Neuroscience
[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013
[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014
[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956
[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010
[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985
[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965
[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981
[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015
[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014
[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999
[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015
[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001
[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996
[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980
[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013
[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013
[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014
[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014
[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach
to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012
[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990
[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987
[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998
Submit your manuscripts athttpwwwhindawicom
Computer Games Technology
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Distributed Sensor Networks
International Journal of
Advances in
FuzzySystems
Hindawi Publishing Corporationhttpwwwhindawicom
Volume 2014
International Journal of
ReconfigurableComputing
Hindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Applied Computational Intelligence and Soft Computing
thinspAdvancesthinspinthinsp
Artificial Intelligence
HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014
Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Journal of
Computer Networks and Communications
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation
httpwwwhindawicom Volume 2014
Advances in
Multimedia
International Journal of
Biomedical Imaging
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
ArtificialNeural Systems
Advances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience
Industrial EngineeringJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Human-ComputerInteraction
Advances in
Computer EngineeringAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience 9
Input Visual stream or Video data 119906(119909 119910 119905)Output Detected MotionConstruct Gaussian window of size119872times119872 denote as 119908for 119905 = Δ119879 2Δ119879 119873Δ119879 do
Divide the screen of frame at time 119905 into multiple119872times119872 blocks 119906119896119897(119905) with overlaps
foreach block 119906119896119897(119905) do
Multiply pixel-wise the block 119906119896119897(119905) with the Gaussian window 119906
119896119897(119905)119908
Take119872times119872 2D FFT of 119906119896119897(119905)119908 denote phase as 120601
119896119897(120596119909 120596119910 119905) and amplitude as 119860
119896119897(120596119909 120596119910 119905)
foreach frequency (120596119909 120596119910) do
Highpass filter temporally 120601119896119897(120596119909 120596119910 119905) denote 1206011015840
119896119897(120596119909 120596119910 119905)
Optionally denoise by 1206011015840119896119897(120596119909 120596119910 119905) larr 120601
1015840
119896119897(120596119909 120596119910 119905)(119860
119896119897(120596119909 120596119910 119905)((1119872
2) sum(120596119909 120596119910)isinD
2 119860119896119897(120596119909 120596119910 119905) + 120598))
endCompute the radon transform (R120601
1015840
119896119897)(120588 120579 119905)
Compute PMI119896119897(119905) according to (38)
if PMI119896119897(119905) gt threshold then
Motion Detected at block (119896 119897) at time 119905Compute direction of motion according to (39)
endend
end
Algorithm 1 Phase-based motion detection algorithm using the FFT
y
120596x
120596y
y
t
120596x
120596y
x
y
x
y
x
x
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
middot middot middot
Divided intooverlapping blocks
Apply window function
2D FFT andextract phase change
Radon transformand extract strength
and orientation of plane
Thresholding
Computephase
change
The phase-baseddetector
Figure 5 Schematic diagram of proposed phase-based motion detection algorithm
10 Computational Intelligence and Neuroscience
in the front and block (23 6) corresponds to an area of thehighway pavement where no motion occurs
Figure 6(f) depicts for each block (119896 119897) the maximumphase change over all (120596
119909 120596119910) isin D2 that is
max(120596119909120596119910)isinD2
10038161003816100381610038161003816100381610038161003816
119889120601119896119897
119889119905
(120596119909 120596119910 1199050)
10038161003816100381610038161003816100381610038161003816
(41)
We observe from the figure that for regions with lowamplitude such as the region depicting the road whenthe normalization constant is absent the derivative of thephase can be noisy For these blocks the maximum of|(119889120601119896119897119889119905)(120596
119909 120596119910 1199050)| over all (120596
119909 120596119910) isin D2 is comparable
to the maximum obtained for blocks that cover the vehiclesin motion
However (29) suggests that the local phase change frommultiple filter pairs centered at the same spatial position(119896 119897) can provide a constraint to robustly estimate motionand its direction Given the block structure employed in thecomputation of the local phase it is natural to utilize phasechange information from multiple sources
Indeed if for a particular block (119896 119897) (119889120601119896119897119889119905)(120596
119909
120596119910 119905) = minusV
119909(119905)120596119909minus V119910(119905)120596119910 then it is easy to see that (119889120601
119896119897
119889119905)(120596119909 120596119910 119905) will be zero on the line V
119909(119905)120596119909+ V119910(119905)120596119910= 0
and have opposite sign on either side of this line For examplein Figures 6(b) and 6(c) (119889120601
119896119897119889119905)(120596
119909 120596119910 1199050) clearly exhibits
this property for blocks that cover a vehicle in motion ThePMI is a tool to evaluate this property
Finally the PMIs for all blocks are shown compactlyin a heat map in Figure 6(e) The figure shows clearly thatthe blocks corresponding to the two moving vehicles have ahigh PMI value while the stationary background areas havea low PMI value allowing one to easily detect motion byemploying simple thresholding (see also the plane ldquoRadontransform and extract strength orientation of planerdquo inFigure 5) In addition the orientation ofmotion in each blockis readily observable even by inspection in Figure 6(b) by aline separating the yellow part and blue part in each blockFurther results about the direction of motion are presentedin Section 4
34 Relationship to Biological Motion Detectors A straight-forward way to implement local motion detectors is to applya complex-valued Gabor receptive field (5) to the video signal119906 and then take the derivative of the phase with respect totime or apply a high-pass filter on the phase to approximatethe derivative
We present here an alternate implementation withoutexplicitly computing the phaseThiswill elucidate the relationbetween the phase-based motion detector presented in theprevious section and some elementary motion detectionmodels used in biology such as theReichardtmotion detector[7] and motion energy detector [9]
Assuming that the local phase 12060100(120596119909 120596119910 119905) is differen-
tiable we have (see also the Appendix)
11988912060100(120596119909 120596119910 119905)
119889119905
=
(119889119887 (120596119909 120596119910 119905) 119889119905) 119886 (120596
119909 120596119910 119905) minus (119889119886 (120596
119909 120596119910 119905) 119889119905) 119887 (120596
119909 120596119910 119905)
[119886 (120596119909 120596119910 119905)]
2
+ [119887 (120596119909 120596119910 119905)]
2 (42)
where 119886(120596119909 120596119910 119905) and 119887(120596
119909 120596119910 119905) are respectively the real
and imaginary parts of 11986000(120596119909 120596119910 119905)11989011989512060100(120596119909120596119910119905)
We notice that the denominator of (42) is the square ofthe local amplitude of 119906 and the numerator is of the formof a second order Volterra kernel This suggests that thetime derivative of the local phase can be viewed as a secondorder Volterra kernel that processes two normalized spatiallyfiltered inputs V
1and V2
We consider an elaborated Reichardt motion detector asshown in Figure 7 It is equipped with a quadrature pair ofGabor filters whose outputs are 119903
1(119905) = 119886(120596
119909 120596119910 119905) and
1199032(119905) = 119887(120596
119909 120596119910 119905) respectively for a particular value of
(120596119909 120596119910) The pair of Gabor filters that provide these outputs
are the real and imaginary parts of119908(119909 119910)119890minus119895(120596119909119909+120596119910119910) It also
consists of a temporal high-pass 1198921(119905) and temporal low-
pass filter 1198922(119905) [9] The output of the elaborated Reichardt
detector follows the diagram in Figure 7 and can be expressedas
(1199032lowast 1198921) (119905) (119903
1lowast 1198922) (119905) minus (119903
1lowast 1198921) (119905) (119903
2lowast 1198922) (119905) (43)
The response can also be characterized by a second orderVolterra kernel We notice the striking similarity between
(43) and the numerator of (42) In fact the phase-basedmotion detector shares some properties with the Reichardtmotion detector For example it is straightforward to see thata single phase-basedmotion detector is tuned to the temporalfrequency of a moving sinusoidal grating
Since the motion energy detector is formally equivalentto an elaborated Reichardt motion detector [9] the structureof the motion energy detector with divisive normalization isalso similar to the phase-based motion detector
4 Exploratory Results
In this section we apply the phase-based motion detectionalgorithm on several video sequences and demonstrate itsefficiency and effectiveness in detecting local motion Themotion detection algorithm is compared to two well-knownbiological motion detectors namely the Reichardt motiondetector and the Barlow-Levick motion detector We thenshow that the detected local motion can be used in motionsegmentation tasks The effectiveness of the segmentationis compared to segmentation using motion informationobtained from a widely used optic flow based algorithmavailable in the literature [4]
Computational Intelligence and Neuroscience 11
(f) (e)
(b)
(a)
(d)
(c)
minus120587 minus1205872 0 1205872
10080604020
y (pi
xel)
100 15050x (pixel)
0020406081
15
10
5
l
01234567+
15
10
5
l
0510152025+
minus6
minus4
minus2
0
2
4
6
181716151413121110
9876543210
l
2 4 6 8 10 12 14 16 18 20 22 240k
minus6minus4minus20246
minus6minus4minus20246
120596x
minus120587 minus1205872 0 1205872
120596x
1205872
0
minus1205872
minus120587
120596y
1205872
0
minus1205872
minus120587
120596y
10 15 20 255k
10 20 255 15k
Figure 6 Evaluating the motion detection algorithm at time 1199050(see also Supplementary Video S1) (a) A still from a 156 times 112-pixel video at
time 1199050 (b) (119889120601
119896119897119889119905)(120596
119909 120596119910 1199050) for all 26times19 blocks each of size 32times32The blocks are concatenated to create a large ldquophase imagerdquo with their
relative neighbors kept (c) (1198891206011211
119889119905)(120596119909 120596119910 1199050) shown in the (120596
119909 120596119910) isin D2 space (d) (119889120601
236119889119905)(120596
119909 120596119910 1199050) shown in the (120596
119909 120596119910) isin D2
space (e) Phase Motion Indicator (PMI) for all blocks at 1199050 Each pixel (119896 119897) is color coded for PMI
119896119897 (f) max
(120596119909 120596119910)isinD2 |(119889120601119896119897119889119905)(120596
119909 120596119910 1199050)|
for all blocks at 1199050 Each pixel (119896 119897) is color coded for this maximum value of block (119896 119897)
g1(t)g2(t) g1(t) g2(t)
times times
minus++
SpatialGaborfilter
SpatialGaborfilter
r1(t) r2(t)
u(x y t)
Figure 7 Diagram of an elaborated Reichardt motion detector Itconsists of a quadrature pair of Gabor filters whose outputs 119903
1and 1199032
provide inputs to the high-pass filters 1198921(119905) and the low-pass filters
1198922(119905)
Table 1 Processing Speeds of the proposed algorithm the Reichardtmotion detector and the Barlow-Levick Motion detector
Screen size Processing capability (frames per second)Proposed Reichardt Barlow-Levick
320 times 240 420 790 800720 times 576 135 685 6901280 times 720 60 274 2931920 times 1080 27 191 194
41 Efficient Parallel Implementation The algorithm wasimplemented in PyCUDA [23] and tested on an NVIDIAGeForce GTX TITAN GPU All computations use singleprecision floating points The processing speeds of the algo-rithm for several screen sizes are listed in Table 1 Clearlythe proposed phase-based motion detection algorithm hasreal-time capability to process video even with full HighDefinition screen size
For comparison we implemented the Reichardt motiondetector and the Barlow-Levick motion detector Theirrespective diagrams are shown in Figure 8 Note that we
12 Computational Intelligence and Neuroscience
ukl(t) uk+1l(t)
g1(t)
g2(t)
g1(t)
g2(t)
+ minus+
timestimes
(a)
minus
minus minus
+
ukl(t) uk+1l(t)
g1(t)g1(t)
g2(t) g2(t)
++
+
(b)
Figure 8 (a) A Reichardt motion detector detecting motion between the two points 119906119896119897and 119906
119896+1119897in the blurred and downsampled video
(b) A Barlow-Levick motion detector detecting motion between the two points 119906119896119897and 119906
119896+1119897in the blurred and downsampled video 119892
1(119905)
denotes a high-pass filter and 1198922(119905) denotes a low-pass filter
moved the high-pass filter ℎ1(119905) to the front of the low-pass fil-
ter ℎ2(119905) This configuration provides a superior performance
to the one in Figure 7 For both the Reichardt and the Barlow-Levick detectors the videos are first blurred by a Gaussianfilter (with the same variance as the Gaussian window in (30)used for the phase-based motion detector) and subsampledat the center of each overlapping block in the phase-basedmotion detector The subsampled video then provides inputsto two 2D arrays of the circuits shown in Figure 8 one forthe horizontal direction and one for the vertical directionThe outputs of the horizontal and vertical motion circuitsform an array of motion vectors that indicate the strengthand direction of motion The three tested motion detectorshave the same number of outputs as a result The processingspeeds of the Reichardt motion detector and the Barlow-Levick motion detector both implemented in PyCUDA andtested on the same GPU are shown in Table 1
Note that the Reichardt motion detector and the Barlow-Levick motion detector are highly efficient due to the sim-plicity of their algorithmsThe phase-basedmotion detectionalgorithm however is a much more sophisticated algorithmand yet it can be implemented in real-time using parallelcomputing devices
The fast GPU implementation is based on the FFTand Matrix-Matrix multiplication It is expected that thoseoperations can be efficiently implemented in hardware forexample FPGA
42 Examples of Phased-BasedMotionDetection Weappliedour motion detection algorithm on video sequences of theChange Detection 2014 dataset [22] that did not exhibitcamera egomotion For these video sequences the standarddeviation of the Gaussian window functions was set to 4
pixels and the block size was chosen to be 32 times 32 pixelsThreshold and normalization parameters were kept the samewith the exception of the ldquothermal videordquo (in order to dealwith larger background noise levels see below) We alsotested the same video sequences using the Reichardt motiondetector and the Barlow-Levick motion detector For theReichardt detector the high-pass filters were chosen as firstorder filters with a time constant of 200 milliseconds and thelow-pass filters were chosen as first order filters with a timeconstant of 300 milliseconds (assuming that the frame rate is50 frames per second)Thresholdwas set to 2 For the Barlow-Levick motion detector the time constant of the first high-pass filters was set to 250 milliseconds The low-pass filterswere the same as in the Reichardt detectorThreshold was setto 2
The first video was taken from a highway surveillancecamera (ldquohighway videordquo) under good illumination con-ditions and high contrast The video had moderate noiseparticularly on the road surface The detected motion isshown in the top left panel of Figure 9 (see SupplementaryVideo S2 for full video) The phase-based motion detectionalgorithm captured both the moving cars and the tree leavesmoving (due to the wind) In the time interval between the9th and 10th second the camera was slightly moved left-and right-wards within 5 frames again possibly due to thewind Movement due to this shift was captured by themotiondetection algorithm and the algorithm was fast enough tocorrectly determine the direction of this movement In thisvideo we already noted that this algorithm suffers from theaperture problem For example in front of the van where along horizontal edge is present the detectedmotion ismostlypointing downwards In addition to moving downwards theedge is alsomoving to the left howeverThis is expected since
Computational Intelligence and Neuroscience 13
(a) (b) (c)
Figure 9 Top motion detection of the ldquohighway videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) Red arrows indicate detected motion and its direction Bottom the contrast of the video was artificiallyreduced by 5-fold and the mean was reduced to 35 of the original The red arrows are duplicated from the motion detection result on theoriginal video as in top Blue arrows are the result of motion detection on the video with reduced contrast If motion is detected both in theoriginal video and in the video with reduced contrast then the arrow is shown in magenta (as a mix of blue and red) (see SupplementaryVideo S2 for full video)
the algorithm only detects motion locally and does not takeinto account the overall shape of any object
For comparison the motion detection results for theReichardt motion detector and the Barlow-Levick motiondetector are shown in the top middle and top right panelof Figure 9 (see Supplementary Video S2 for full video) TheReichardt detector performed relatively well when vehiclesare moving faster but the direction it predicts for vehiclesmoving slower for example on the back of the image is notaccurate In addition motion is still detected in some partsof the screen where the vehicles have just passed by Thedetection result for the Barlow-Levick motion detector waspoorer In particular the response to OFF edge movement isalways the opposite to the actual movement direction
We then squeezed the range of the screen intensity from[0 1] to [02 04] resulting in a video with a lower meanluminance and lower contrast The motion detection resultson the low-contrast video are shown in the bottom 3 panelsof Figure 9 (see Supplementary Video S2 for full video) Forreference the motion detection results for the original videoare shown in red arrows Motion detected in the low-contrastvideo is shown in blue arrows if no motion is detected forthe block in the original video If motion is detected in boththe original and the low-contrast video the arrows are shownin magenta Figure 9 clearly shows that while the motiondetection performance is degraded for all three detectors thephase-basedmotion detector performed still quite well in thelow-contrast video detecting most of the moving vehicles
The other two detectors missed many of the blocks wheremotion was detected in the original movie
To quantify how well each detector works under differentcontrast conditions we computed the ratio of unthresholdedoutput values of eachmotion detector between lower contrastand full contrast video For example in the case of phase-basedmotion detector we computed for each block the ratiobetween the PMI index for the lower contrast video and thatfor the full contrast videoThe ratios are then averaged acrossall blocks where motion is detected in the full contrast videoThis average for different contrasts is shown in Figure 10 bythe blue curve In the ideal case when the normalizationconstant 120598 is 0 the phase detector should produce invariantresponses to videos with different contrast The curve shownhere is mainly due to a nonzero 120598The ratios for the Reichardtmotion detector and the Barlow-Levick motion detectorare computed similarly and are shown in red and yellowrespectively It is clear that the phase-based motion detectorhas a superior performance across a range of contrast valuesAt 20 contrast the phase-based detector still has 50 ofthe PMI index value for full contrast video As expectedthe response of Barlow-Levick motion detector is linear withrespect to contrast and the Reichardt motion detector has aquadratic relation to contrast For a fixed threshold value alarger ratio equates to amore consistent performance at lowercontrast
The second video was captured by a surveillance camerain a train station (ldquotrain station videordquo) The video was
14 Computational Intelligence and Neuroscience
Phase-basedReichardtBarlow-Levick
0
02
04
06
08
1
Ratio
40 60 80200 100Contrast ()
Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined
under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm
The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)
The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression
With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)
43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient
objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area
For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold
We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually
We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm
We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)
These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm
5 Discussion
Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes
Computational Intelligence and Neuroscience 15
(a) (b) (c)
Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)
(a) (b) (c)
Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)
by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to
models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]
We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in
16 Computational Intelligence and Neuroscience
(a) (b) (c)
Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)
Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)
phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in
Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)
modeling in vivo biological vision systems This is consistentwith other recent findings [15]
Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure
Computational Intelligence and Neuroscience 17
(a) (b)
Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)
(a) (b)
Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)
(a) (b)
Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)
Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase
itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third
18 Computational Intelligence and Neuroscience
(a) (b)
Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)
instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection
The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions
Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors
We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well
We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It
can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes
Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems
Appendix
The Derivative of the Local Phase
Let
119886 (120596119909 120596119910 119905)
= int
R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596
119909119909 + 120596119910119910) 119889119909 119889119910
119887 (120596119909 120596119910 119905)
= minusint
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
(A1)
and therefore
11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)
= 119886 (120596119909 120596119910 119905) + 119895119887 (120596
119909 120596119910 119905)
(A2)
The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596
119909
120596119910 119905) 119886(120596
119909 120596119910 119905)) and therefore
11988912060100
119889119905
=
119886 (119889119887119889119905) minus 119887 (119889119886119889119905)
[119886]2+ [119887]2
(A3)
Computational Intelligence and Neuroscience 19
Now
119889119886
119889119905
(120596119909 120596119910 119905)
= int
R2
120597119906
120597119905
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
(A4)
For a purely translational signal with instantaneous spatialshift given by (119904
119909(119905) 119904119910(119905)) within the support of the window
we have
119889119886
119889119905
(120596119909 120596119910 119905)
= minusint
R2
119889119904119909
119889119905
(119905)
120597119906
120597119909
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus int
R2
119889119904119910
119889119905
(119905)
120597119906
120597119910
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119909
119889119904119909
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119910
119889119904119910
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199033=1199033(120596119909120596119910119905)
+ 120596119909
119889119904119909
119889119905
119887
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199034=1199034(120596119909120596119910119905)
+ 120596119910
119889119904119910
119889119905
119887
(A5)
Similarly
119889119887
119889119905
(120596119909 120596119910 119905) = minus
119889119904119909
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199035=1199035(120596119909120596119910119905)
minus 120596119909
119889119904119909
119889119905
119886 minus
119889119904119910
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199036=1199036(120596119909120596119910119905)
minus 120596119910
119889119904119910
119889119905
119886
(A6)
Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to
11988912060100
119889119905
(120596119909 120596119910 119905) =
minus (119889119904119909119889119905) [119903
3119887 + 1199035119886 + 120596119909 [
119886]2+ 120596119909 [
119887]2] minus (119889119904
119910119889119905) [119903
4119887 + 1199036119886 + 120596119910 [
119886]2+ 120596119910 [
119887]2]
[119886]2+ [119887]2
= minus
119889119904119909
119889119905
120596119909minus
119889119904119910
119889119905
120596119910minus
119889119904119909
119889119905
sdot
1199033119887 + 1199035119886
[119886]2+ [119887]2minus
119889119904119910
119889119905
sdot
1199034119887 + 1199036119886
[119886]2+ [119887]2
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
v00(120596119909120596119910119905)
(A7)
Disclosure
The authorrsquos names are alphabetically listed
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgment
The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232
References
[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010
[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010
[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014
[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010
20 Computational Intelligence and Neuroscience
[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013
[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014
[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956
[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010
[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985
[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965
[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981
[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015
[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014
[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999
[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015
[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001
[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996
[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980
[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013
[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013
[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014
[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014
[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach
to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012
[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990
[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987
[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998
Submit your manuscripts athttpwwwhindawicom
Computer Games Technology
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Distributed Sensor Networks
International Journal of
Advances in
FuzzySystems
Hindawi Publishing Corporationhttpwwwhindawicom
Volume 2014
International Journal of
ReconfigurableComputing
Hindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Applied Computational Intelligence and Soft Computing
thinspAdvancesthinspinthinsp
Artificial Intelligence
HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014
Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Journal of
Computer Networks and Communications
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation
httpwwwhindawicom Volume 2014
Advances in
Multimedia
International Journal of
Biomedical Imaging
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
ArtificialNeural Systems
Advances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience
Industrial EngineeringJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Human-ComputerInteraction
Advances in
Computer EngineeringAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
10 Computational Intelligence and Neuroscience
in the front and block (23 6) corresponds to an area of thehighway pavement where no motion occurs
Figure 6(f) depicts for each block (119896 119897) the maximumphase change over all (120596
119909 120596119910) isin D2 that is
max(120596119909120596119910)isinD2
10038161003816100381610038161003816100381610038161003816
119889120601119896119897
119889119905
(120596119909 120596119910 1199050)
10038161003816100381610038161003816100381610038161003816
(41)
We observe from the figure that for regions with lowamplitude such as the region depicting the road whenthe normalization constant is absent the derivative of thephase can be noisy For these blocks the maximum of|(119889120601119896119897119889119905)(120596
119909 120596119910 1199050)| over all (120596
119909 120596119910) isin D2 is comparable
to the maximum obtained for blocks that cover the vehiclesin motion
However (29) suggests that the local phase change frommultiple filter pairs centered at the same spatial position(119896 119897) can provide a constraint to robustly estimate motionand its direction Given the block structure employed in thecomputation of the local phase it is natural to utilize phasechange information from multiple sources
Indeed if for a particular block (119896 119897) (119889120601119896119897119889119905)(120596
119909
120596119910 119905) = minusV
119909(119905)120596119909minus V119910(119905)120596119910 then it is easy to see that (119889120601
119896119897
119889119905)(120596119909 120596119910 119905) will be zero on the line V
119909(119905)120596119909+ V119910(119905)120596119910= 0
and have opposite sign on either side of this line For examplein Figures 6(b) and 6(c) (119889120601
119896119897119889119905)(120596
119909 120596119910 1199050) clearly exhibits
this property for blocks that cover a vehicle in motion ThePMI is a tool to evaluate this property
Finally the PMIs for all blocks are shown compactlyin a heat map in Figure 6(e) The figure shows clearly thatthe blocks corresponding to the two moving vehicles have ahigh PMI value while the stationary background areas havea low PMI value allowing one to easily detect motion byemploying simple thresholding (see also the plane ldquoRadontransform and extract strength orientation of planerdquo inFigure 5) In addition the orientation ofmotion in each blockis readily observable even by inspection in Figure 6(b) by aline separating the yellow part and blue part in each blockFurther results about the direction of motion are presentedin Section 4
34 Relationship to Biological Motion Detectors A straight-forward way to implement local motion detectors is to applya complex-valued Gabor receptive field (5) to the video signal119906 and then take the derivative of the phase with respect totime or apply a high-pass filter on the phase to approximatethe derivative
We present here an alternate implementation withoutexplicitly computing the phaseThiswill elucidate the relationbetween the phase-based motion detector presented in theprevious section and some elementary motion detectionmodels used in biology such as theReichardtmotion detector[7] and motion energy detector [9]
Assuming that the local phase 12060100(120596119909 120596119910 119905) is differen-
tiable we have (see also the Appendix)
11988912060100(120596119909 120596119910 119905)
119889119905
=
(119889119887 (120596119909 120596119910 119905) 119889119905) 119886 (120596
119909 120596119910 119905) minus (119889119886 (120596
119909 120596119910 119905) 119889119905) 119887 (120596
119909 120596119910 119905)
[119886 (120596119909 120596119910 119905)]
2
+ [119887 (120596119909 120596119910 119905)]
2 (42)
where 119886(120596119909 120596119910 119905) and 119887(120596
119909 120596119910 119905) are respectively the real
and imaginary parts of 11986000(120596119909 120596119910 119905)11989011989512060100(120596119909120596119910119905)
We notice that the denominator of (42) is the square ofthe local amplitude of 119906 and the numerator is of the formof a second order Volterra kernel This suggests that thetime derivative of the local phase can be viewed as a secondorder Volterra kernel that processes two normalized spatiallyfiltered inputs V
1and V2
We consider an elaborated Reichardt motion detector asshown in Figure 7 It is equipped with a quadrature pair ofGabor filters whose outputs are 119903
1(119905) = 119886(120596
119909 120596119910 119905) and
1199032(119905) = 119887(120596
119909 120596119910 119905) respectively for a particular value of
(120596119909 120596119910) The pair of Gabor filters that provide these outputs
are the real and imaginary parts of119908(119909 119910)119890minus119895(120596119909119909+120596119910119910) It also
consists of a temporal high-pass 1198921(119905) and temporal low-
pass filter 1198922(119905) [9] The output of the elaborated Reichardt
detector follows the diagram in Figure 7 and can be expressedas
(1199032lowast 1198921) (119905) (119903
1lowast 1198922) (119905) minus (119903
1lowast 1198921) (119905) (119903
2lowast 1198922) (119905) (43)
The response can also be characterized by a second orderVolterra kernel We notice the striking similarity between
(43) and the numerator of (42) In fact the phase-basedmotion detector shares some properties with the Reichardtmotion detector For example it is straightforward to see thata single phase-basedmotion detector is tuned to the temporalfrequency of a moving sinusoidal grating
Since the motion energy detector is formally equivalentto an elaborated Reichardt motion detector [9] the structureof the motion energy detector with divisive normalization isalso similar to the phase-based motion detector
4 Exploratory Results
In this section we apply the phase-based motion detectionalgorithm on several video sequences and demonstrate itsefficiency and effectiveness in detecting local motion Themotion detection algorithm is compared to two well-knownbiological motion detectors namely the Reichardt motiondetector and the Barlow-Levick motion detector We thenshow that the detected local motion can be used in motionsegmentation tasks The effectiveness of the segmentationis compared to segmentation using motion informationobtained from a widely used optic flow based algorithmavailable in the literature [4]
Computational Intelligence and Neuroscience 11
(f) (e)
(b)
(a)
(d)
(c)
minus120587 minus1205872 0 1205872
10080604020
y (pi
xel)
100 15050x (pixel)
0020406081
15
10
5
l
01234567+
15
10
5
l
0510152025+
minus6
minus4
minus2
0
2
4
6
181716151413121110
9876543210
l
2 4 6 8 10 12 14 16 18 20 22 240k
minus6minus4minus20246
minus6minus4minus20246
120596x
minus120587 minus1205872 0 1205872
120596x
1205872
0
minus1205872
minus120587
120596y
1205872
0
minus1205872
minus120587
120596y
10 15 20 255k
10 20 255 15k
Figure 6 Evaluating the motion detection algorithm at time 1199050(see also Supplementary Video S1) (a) A still from a 156 times 112-pixel video at
time 1199050 (b) (119889120601
119896119897119889119905)(120596
119909 120596119910 1199050) for all 26times19 blocks each of size 32times32The blocks are concatenated to create a large ldquophase imagerdquo with their
relative neighbors kept (c) (1198891206011211
119889119905)(120596119909 120596119910 1199050) shown in the (120596
119909 120596119910) isin D2 space (d) (119889120601
236119889119905)(120596
119909 120596119910 1199050) shown in the (120596
119909 120596119910) isin D2
space (e) Phase Motion Indicator (PMI) for all blocks at 1199050 Each pixel (119896 119897) is color coded for PMI
119896119897 (f) max
(120596119909 120596119910)isinD2 |(119889120601119896119897119889119905)(120596
119909 120596119910 1199050)|
for all blocks at 1199050 Each pixel (119896 119897) is color coded for this maximum value of block (119896 119897)
g1(t)g2(t) g1(t) g2(t)
times times
minus++
SpatialGaborfilter
SpatialGaborfilter
r1(t) r2(t)
u(x y t)
Figure 7 Diagram of an elaborated Reichardt motion detector Itconsists of a quadrature pair of Gabor filters whose outputs 119903
1and 1199032
provide inputs to the high-pass filters 1198921(119905) and the low-pass filters
1198922(119905)
Table 1 Processing Speeds of the proposed algorithm the Reichardtmotion detector and the Barlow-Levick Motion detector
Screen size Processing capability (frames per second)Proposed Reichardt Barlow-Levick
320 times 240 420 790 800720 times 576 135 685 6901280 times 720 60 274 2931920 times 1080 27 191 194
41 Efficient Parallel Implementation The algorithm wasimplemented in PyCUDA [23] and tested on an NVIDIAGeForce GTX TITAN GPU All computations use singleprecision floating points The processing speeds of the algo-rithm for several screen sizes are listed in Table 1 Clearlythe proposed phase-based motion detection algorithm hasreal-time capability to process video even with full HighDefinition screen size
For comparison we implemented the Reichardt motiondetector and the Barlow-Levick motion detector Theirrespective diagrams are shown in Figure 8 Note that we
12 Computational Intelligence and Neuroscience
ukl(t) uk+1l(t)
g1(t)
g2(t)
g1(t)
g2(t)
+ minus+
timestimes
(a)
minus
minus minus
+
ukl(t) uk+1l(t)
g1(t)g1(t)
g2(t) g2(t)
++
+
(b)
Figure 8 (a) A Reichardt motion detector detecting motion between the two points 119906119896119897and 119906
119896+1119897in the blurred and downsampled video
(b) A Barlow-Levick motion detector detecting motion between the two points 119906119896119897and 119906
119896+1119897in the blurred and downsampled video 119892
1(119905)
denotes a high-pass filter and 1198922(119905) denotes a low-pass filter
moved the high-pass filter ℎ1(119905) to the front of the low-pass fil-
ter ℎ2(119905) This configuration provides a superior performance
to the one in Figure 7 For both the Reichardt and the Barlow-Levick detectors the videos are first blurred by a Gaussianfilter (with the same variance as the Gaussian window in (30)used for the phase-based motion detector) and subsampledat the center of each overlapping block in the phase-basedmotion detector The subsampled video then provides inputsto two 2D arrays of the circuits shown in Figure 8 one forthe horizontal direction and one for the vertical directionThe outputs of the horizontal and vertical motion circuitsform an array of motion vectors that indicate the strengthand direction of motion The three tested motion detectorshave the same number of outputs as a result The processingspeeds of the Reichardt motion detector and the Barlow-Levick motion detector both implemented in PyCUDA andtested on the same GPU are shown in Table 1
Note that the Reichardt motion detector and the Barlow-Levick motion detector are highly efficient due to the sim-plicity of their algorithmsThe phase-basedmotion detectionalgorithm however is a much more sophisticated algorithmand yet it can be implemented in real-time using parallelcomputing devices
The fast GPU implementation is based on the FFTand Matrix-Matrix multiplication It is expected that thoseoperations can be efficiently implemented in hardware forexample FPGA
42 Examples of Phased-BasedMotionDetection Weappliedour motion detection algorithm on video sequences of theChange Detection 2014 dataset [22] that did not exhibitcamera egomotion For these video sequences the standarddeviation of the Gaussian window functions was set to 4
pixels and the block size was chosen to be 32 times 32 pixelsThreshold and normalization parameters were kept the samewith the exception of the ldquothermal videordquo (in order to dealwith larger background noise levels see below) We alsotested the same video sequences using the Reichardt motiondetector and the Barlow-Levick motion detector For theReichardt detector the high-pass filters were chosen as firstorder filters with a time constant of 200 milliseconds and thelow-pass filters were chosen as first order filters with a timeconstant of 300 milliseconds (assuming that the frame rate is50 frames per second)Thresholdwas set to 2 For the Barlow-Levick motion detector the time constant of the first high-pass filters was set to 250 milliseconds The low-pass filterswere the same as in the Reichardt detectorThreshold was setto 2
The first video was taken from a highway surveillancecamera (ldquohighway videordquo) under good illumination con-ditions and high contrast The video had moderate noiseparticularly on the road surface The detected motion isshown in the top left panel of Figure 9 (see SupplementaryVideo S2 for full video) The phase-based motion detectionalgorithm captured both the moving cars and the tree leavesmoving (due to the wind) In the time interval between the9th and 10th second the camera was slightly moved left-and right-wards within 5 frames again possibly due to thewind Movement due to this shift was captured by themotiondetection algorithm and the algorithm was fast enough tocorrectly determine the direction of this movement In thisvideo we already noted that this algorithm suffers from theaperture problem For example in front of the van where along horizontal edge is present the detectedmotion ismostlypointing downwards In addition to moving downwards theedge is alsomoving to the left howeverThis is expected since
Computational Intelligence and Neuroscience 13
(a) (b) (c)
Figure 9 Top motion detection of the ldquohighway videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) Red arrows indicate detected motion and its direction Bottom the contrast of the video was artificiallyreduced by 5-fold and the mean was reduced to 35 of the original The red arrows are duplicated from the motion detection result on theoriginal video as in top Blue arrows are the result of motion detection on the video with reduced contrast If motion is detected both in theoriginal video and in the video with reduced contrast then the arrow is shown in magenta (as a mix of blue and red) (see SupplementaryVideo S2 for full video)
the algorithm only detects motion locally and does not takeinto account the overall shape of any object
For comparison the motion detection results for theReichardt motion detector and the Barlow-Levick motiondetector are shown in the top middle and top right panelof Figure 9 (see Supplementary Video S2 for full video) TheReichardt detector performed relatively well when vehiclesare moving faster but the direction it predicts for vehiclesmoving slower for example on the back of the image is notaccurate In addition motion is still detected in some partsof the screen where the vehicles have just passed by Thedetection result for the Barlow-Levick motion detector waspoorer In particular the response to OFF edge movement isalways the opposite to the actual movement direction
We then squeezed the range of the screen intensity from[0 1] to [02 04] resulting in a video with a lower meanluminance and lower contrast The motion detection resultson the low-contrast video are shown in the bottom 3 panelsof Figure 9 (see Supplementary Video S2 for full video) Forreference the motion detection results for the original videoare shown in red arrows Motion detected in the low-contrastvideo is shown in blue arrows if no motion is detected forthe block in the original video If motion is detected in boththe original and the low-contrast video the arrows are shownin magenta Figure 9 clearly shows that while the motiondetection performance is degraded for all three detectors thephase-basedmotion detector performed still quite well in thelow-contrast video detecting most of the moving vehicles
The other two detectors missed many of the blocks wheremotion was detected in the original movie
To quantify how well each detector works under differentcontrast conditions we computed the ratio of unthresholdedoutput values of eachmotion detector between lower contrastand full contrast video For example in the case of phase-basedmotion detector we computed for each block the ratiobetween the PMI index for the lower contrast video and thatfor the full contrast videoThe ratios are then averaged acrossall blocks where motion is detected in the full contrast videoThis average for different contrasts is shown in Figure 10 bythe blue curve In the ideal case when the normalizationconstant 120598 is 0 the phase detector should produce invariantresponses to videos with different contrast The curve shownhere is mainly due to a nonzero 120598The ratios for the Reichardtmotion detector and the Barlow-Levick motion detectorare computed similarly and are shown in red and yellowrespectively It is clear that the phase-based motion detectorhas a superior performance across a range of contrast valuesAt 20 contrast the phase-based detector still has 50 ofthe PMI index value for full contrast video As expectedthe response of Barlow-Levick motion detector is linear withrespect to contrast and the Reichardt motion detector has aquadratic relation to contrast For a fixed threshold value alarger ratio equates to amore consistent performance at lowercontrast
The second video was captured by a surveillance camerain a train station (ldquotrain station videordquo) The video was
14 Computational Intelligence and Neuroscience
Phase-basedReichardtBarlow-Levick
0
02
04
06
08
1
Ratio
40 60 80200 100Contrast ()
Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined
under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm
The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)
The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression
With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)
43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient
objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area
For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold
We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually
We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm
We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)
These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm
5 Discussion
Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes
Computational Intelligence and Neuroscience 15
(a) (b) (c)
Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)
(a) (b) (c)
Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)
by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to
models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]
We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in
16 Computational Intelligence and Neuroscience
(a) (b) (c)
Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)
Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)
phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in
Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)
modeling in vivo biological vision systems This is consistentwith other recent findings [15]
Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure
Computational Intelligence and Neuroscience 17
(a) (b)
Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)
(a) (b)
Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)
(a) (b)
Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)
Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase
itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third
18 Computational Intelligence and Neuroscience
(a) (b)
Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)
instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection
The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions
Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors
We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well
We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It
can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes
Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems
Appendix
The Derivative of the Local Phase
Let
119886 (120596119909 120596119910 119905)
= int
R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596
119909119909 + 120596119910119910) 119889119909 119889119910
119887 (120596119909 120596119910 119905)
= minusint
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
(A1)
and therefore
11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)
= 119886 (120596119909 120596119910 119905) + 119895119887 (120596
119909 120596119910 119905)
(A2)
The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596
119909
120596119910 119905) 119886(120596
119909 120596119910 119905)) and therefore
11988912060100
119889119905
=
119886 (119889119887119889119905) minus 119887 (119889119886119889119905)
[119886]2+ [119887]2
(A3)
Computational Intelligence and Neuroscience 19
Now
119889119886
119889119905
(120596119909 120596119910 119905)
= int
R2
120597119906
120597119905
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
(A4)
For a purely translational signal with instantaneous spatialshift given by (119904
119909(119905) 119904119910(119905)) within the support of the window
we have
119889119886
119889119905
(120596119909 120596119910 119905)
= minusint
R2
119889119904119909
119889119905
(119905)
120597119906
120597119909
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus int
R2
119889119904119910
119889119905
(119905)
120597119906
120597119910
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119909
119889119904119909
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119910
119889119904119910
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199033=1199033(120596119909120596119910119905)
+ 120596119909
119889119904119909
119889119905
119887
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199034=1199034(120596119909120596119910119905)
+ 120596119910
119889119904119910
119889119905
119887
(A5)
Similarly
119889119887
119889119905
(120596119909 120596119910 119905) = minus
119889119904119909
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199035=1199035(120596119909120596119910119905)
minus 120596119909
119889119904119909
119889119905
119886 minus
119889119904119910
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199036=1199036(120596119909120596119910119905)
minus 120596119910
119889119904119910
119889119905
119886
(A6)
Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to
11988912060100
119889119905
(120596119909 120596119910 119905) =
minus (119889119904119909119889119905) [119903
3119887 + 1199035119886 + 120596119909 [
119886]2+ 120596119909 [
119887]2] minus (119889119904
119910119889119905) [119903
4119887 + 1199036119886 + 120596119910 [
119886]2+ 120596119910 [
119887]2]
[119886]2+ [119887]2
= minus
119889119904119909
119889119905
120596119909minus
119889119904119910
119889119905
120596119910minus
119889119904119909
119889119905
sdot
1199033119887 + 1199035119886
[119886]2+ [119887]2minus
119889119904119910
119889119905
sdot
1199034119887 + 1199036119886
[119886]2+ [119887]2
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
v00(120596119909120596119910119905)
(A7)
Disclosure
The authorrsquos names are alphabetically listed
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgment
The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232
References
[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010
[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010
[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014
[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010
20 Computational Intelligence and Neuroscience
[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013
[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014
[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956
[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010
[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985
[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965
[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981
[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015
[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014
[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999
[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015
[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001
[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996
[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980
[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013
[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013
[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014
[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014
[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach
to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012
[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990
[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987
[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998
Submit your manuscripts athttpwwwhindawicom
Computer Games Technology
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Distributed Sensor Networks
International Journal of
Advances in
FuzzySystems
Hindawi Publishing Corporationhttpwwwhindawicom
Volume 2014
International Journal of
ReconfigurableComputing
Hindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Applied Computational Intelligence and Soft Computing
thinspAdvancesthinspinthinsp
Artificial Intelligence
HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014
Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Journal of
Computer Networks and Communications
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation
httpwwwhindawicom Volume 2014
Advances in
Multimedia
International Journal of
Biomedical Imaging
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
ArtificialNeural Systems
Advances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience
Industrial EngineeringJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Human-ComputerInteraction
Advances in
Computer EngineeringAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience 11
(f) (e)
(b)
(a)
(d)
(c)
minus120587 minus1205872 0 1205872
10080604020
y (pi
xel)
100 15050x (pixel)
0020406081
15
10
5
l
01234567+
15
10
5
l
0510152025+
minus6
minus4
minus2
0
2
4
6
181716151413121110
9876543210
l
2 4 6 8 10 12 14 16 18 20 22 240k
minus6minus4minus20246
minus6minus4minus20246
120596x
minus120587 minus1205872 0 1205872
120596x
1205872
0
minus1205872
minus120587
120596y
1205872
0
minus1205872
minus120587
120596y
10 15 20 255k
10 20 255 15k
Figure 6 Evaluating the motion detection algorithm at time 1199050(see also Supplementary Video S1) (a) A still from a 156 times 112-pixel video at
time 1199050 (b) (119889120601
119896119897119889119905)(120596
119909 120596119910 1199050) for all 26times19 blocks each of size 32times32The blocks are concatenated to create a large ldquophase imagerdquo with their
relative neighbors kept (c) (1198891206011211
119889119905)(120596119909 120596119910 1199050) shown in the (120596
119909 120596119910) isin D2 space (d) (119889120601
236119889119905)(120596
119909 120596119910 1199050) shown in the (120596
119909 120596119910) isin D2
space (e) Phase Motion Indicator (PMI) for all blocks at 1199050 Each pixel (119896 119897) is color coded for PMI
119896119897 (f) max
(120596119909 120596119910)isinD2 |(119889120601119896119897119889119905)(120596
119909 120596119910 1199050)|
for all blocks at 1199050 Each pixel (119896 119897) is color coded for this maximum value of block (119896 119897)
g1(t)g2(t) g1(t) g2(t)
times times
minus++
SpatialGaborfilter
SpatialGaborfilter
r1(t) r2(t)
u(x y t)
Figure 7 Diagram of an elaborated Reichardt motion detector Itconsists of a quadrature pair of Gabor filters whose outputs 119903
1and 1199032
provide inputs to the high-pass filters 1198921(119905) and the low-pass filters
1198922(119905)
Table 1 Processing Speeds of the proposed algorithm the Reichardtmotion detector and the Barlow-Levick Motion detector
Screen size Processing capability (frames per second)Proposed Reichardt Barlow-Levick
320 times 240 420 790 800720 times 576 135 685 6901280 times 720 60 274 2931920 times 1080 27 191 194
41 Efficient Parallel Implementation The algorithm wasimplemented in PyCUDA [23] and tested on an NVIDIAGeForce GTX TITAN GPU All computations use singleprecision floating points The processing speeds of the algo-rithm for several screen sizes are listed in Table 1 Clearlythe proposed phase-based motion detection algorithm hasreal-time capability to process video even with full HighDefinition screen size
For comparison we implemented the Reichardt motiondetector and the Barlow-Levick motion detector Theirrespective diagrams are shown in Figure 8 Note that we
12 Computational Intelligence and Neuroscience
ukl(t) uk+1l(t)
g1(t)
g2(t)
g1(t)
g2(t)
+ minus+
timestimes
(a)
minus
minus minus
+
ukl(t) uk+1l(t)
g1(t)g1(t)
g2(t) g2(t)
++
+
(b)
Figure 8 (a) A Reichardt motion detector detecting motion between the two points 119906119896119897and 119906
119896+1119897in the blurred and downsampled video
(b) A Barlow-Levick motion detector detecting motion between the two points 119906119896119897and 119906
119896+1119897in the blurred and downsampled video 119892
1(119905)
denotes a high-pass filter and 1198922(119905) denotes a low-pass filter
moved the high-pass filter ℎ1(119905) to the front of the low-pass fil-
ter ℎ2(119905) This configuration provides a superior performance
to the one in Figure 7 For both the Reichardt and the Barlow-Levick detectors the videos are first blurred by a Gaussianfilter (with the same variance as the Gaussian window in (30)used for the phase-based motion detector) and subsampledat the center of each overlapping block in the phase-basedmotion detector The subsampled video then provides inputsto two 2D arrays of the circuits shown in Figure 8 one forthe horizontal direction and one for the vertical directionThe outputs of the horizontal and vertical motion circuitsform an array of motion vectors that indicate the strengthand direction of motion The three tested motion detectorshave the same number of outputs as a result The processingspeeds of the Reichardt motion detector and the Barlow-Levick motion detector both implemented in PyCUDA andtested on the same GPU are shown in Table 1
Note that the Reichardt motion detector and the Barlow-Levick motion detector are highly efficient due to the sim-plicity of their algorithmsThe phase-basedmotion detectionalgorithm however is a much more sophisticated algorithmand yet it can be implemented in real-time using parallelcomputing devices
The fast GPU implementation is based on the FFTand Matrix-Matrix multiplication It is expected that thoseoperations can be efficiently implemented in hardware forexample FPGA
42 Examples of Phased-BasedMotionDetection Weappliedour motion detection algorithm on video sequences of theChange Detection 2014 dataset [22] that did not exhibitcamera egomotion For these video sequences the standarddeviation of the Gaussian window functions was set to 4
pixels and the block size was chosen to be 32 times 32 pixelsThreshold and normalization parameters were kept the samewith the exception of the ldquothermal videordquo (in order to dealwith larger background noise levels see below) We alsotested the same video sequences using the Reichardt motiondetector and the Barlow-Levick motion detector For theReichardt detector the high-pass filters were chosen as firstorder filters with a time constant of 200 milliseconds and thelow-pass filters were chosen as first order filters with a timeconstant of 300 milliseconds (assuming that the frame rate is50 frames per second)Thresholdwas set to 2 For the Barlow-Levick motion detector the time constant of the first high-pass filters was set to 250 milliseconds The low-pass filterswere the same as in the Reichardt detectorThreshold was setto 2
The first video was taken from a highway surveillancecamera (ldquohighway videordquo) under good illumination con-ditions and high contrast The video had moderate noiseparticularly on the road surface The detected motion isshown in the top left panel of Figure 9 (see SupplementaryVideo S2 for full video) The phase-based motion detectionalgorithm captured both the moving cars and the tree leavesmoving (due to the wind) In the time interval between the9th and 10th second the camera was slightly moved left-and right-wards within 5 frames again possibly due to thewind Movement due to this shift was captured by themotiondetection algorithm and the algorithm was fast enough tocorrectly determine the direction of this movement In thisvideo we already noted that this algorithm suffers from theaperture problem For example in front of the van where along horizontal edge is present the detectedmotion ismostlypointing downwards In addition to moving downwards theedge is alsomoving to the left howeverThis is expected since
Computational Intelligence and Neuroscience 13
(a) (b) (c)
Figure 9 Top motion detection of the ldquohighway videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) Red arrows indicate detected motion and its direction Bottom the contrast of the video was artificiallyreduced by 5-fold and the mean was reduced to 35 of the original The red arrows are duplicated from the motion detection result on theoriginal video as in top Blue arrows are the result of motion detection on the video with reduced contrast If motion is detected both in theoriginal video and in the video with reduced contrast then the arrow is shown in magenta (as a mix of blue and red) (see SupplementaryVideo S2 for full video)
the algorithm only detects motion locally and does not takeinto account the overall shape of any object
For comparison the motion detection results for theReichardt motion detector and the Barlow-Levick motiondetector are shown in the top middle and top right panelof Figure 9 (see Supplementary Video S2 for full video) TheReichardt detector performed relatively well when vehiclesare moving faster but the direction it predicts for vehiclesmoving slower for example on the back of the image is notaccurate In addition motion is still detected in some partsof the screen where the vehicles have just passed by Thedetection result for the Barlow-Levick motion detector waspoorer In particular the response to OFF edge movement isalways the opposite to the actual movement direction
We then squeezed the range of the screen intensity from[0 1] to [02 04] resulting in a video with a lower meanluminance and lower contrast The motion detection resultson the low-contrast video are shown in the bottom 3 panelsof Figure 9 (see Supplementary Video S2 for full video) Forreference the motion detection results for the original videoare shown in red arrows Motion detected in the low-contrastvideo is shown in blue arrows if no motion is detected forthe block in the original video If motion is detected in boththe original and the low-contrast video the arrows are shownin magenta Figure 9 clearly shows that while the motiondetection performance is degraded for all three detectors thephase-basedmotion detector performed still quite well in thelow-contrast video detecting most of the moving vehicles
The other two detectors missed many of the blocks wheremotion was detected in the original movie
To quantify how well each detector works under differentcontrast conditions we computed the ratio of unthresholdedoutput values of eachmotion detector between lower contrastand full contrast video For example in the case of phase-basedmotion detector we computed for each block the ratiobetween the PMI index for the lower contrast video and thatfor the full contrast videoThe ratios are then averaged acrossall blocks where motion is detected in the full contrast videoThis average for different contrasts is shown in Figure 10 bythe blue curve In the ideal case when the normalizationconstant 120598 is 0 the phase detector should produce invariantresponses to videos with different contrast The curve shownhere is mainly due to a nonzero 120598The ratios for the Reichardtmotion detector and the Barlow-Levick motion detectorare computed similarly and are shown in red and yellowrespectively It is clear that the phase-based motion detectorhas a superior performance across a range of contrast valuesAt 20 contrast the phase-based detector still has 50 ofthe PMI index value for full contrast video As expectedthe response of Barlow-Levick motion detector is linear withrespect to contrast and the Reichardt motion detector has aquadratic relation to contrast For a fixed threshold value alarger ratio equates to amore consistent performance at lowercontrast
The second video was captured by a surveillance camerain a train station (ldquotrain station videordquo) The video was
14 Computational Intelligence and Neuroscience
Phase-basedReichardtBarlow-Levick
0
02
04
06
08
1
Ratio
40 60 80200 100Contrast ()
Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined
under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm
The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)
The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression
With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)
43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient
objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area
For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold
We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually
We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm
We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)
These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm
5 Discussion
Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes
Computational Intelligence and Neuroscience 15
(a) (b) (c)
Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)
(a) (b) (c)
Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)
by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to
models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]
We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in
16 Computational Intelligence and Neuroscience
(a) (b) (c)
Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)
Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)
phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in
Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)
modeling in vivo biological vision systems This is consistentwith other recent findings [15]
Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure
Computational Intelligence and Neuroscience 17
(a) (b)
Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)
(a) (b)
Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)
(a) (b)
Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)
Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase
itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third
18 Computational Intelligence and Neuroscience
(a) (b)
Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)
instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection
The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions
Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors
We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well
We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It
can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes
Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems
Appendix
The Derivative of the Local Phase
Let
119886 (120596119909 120596119910 119905)
= int
R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596
119909119909 + 120596119910119910) 119889119909 119889119910
119887 (120596119909 120596119910 119905)
= minusint
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
(A1)
and therefore
11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)
= 119886 (120596119909 120596119910 119905) + 119895119887 (120596
119909 120596119910 119905)
(A2)
The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596
119909
120596119910 119905) 119886(120596
119909 120596119910 119905)) and therefore
11988912060100
119889119905
=
119886 (119889119887119889119905) minus 119887 (119889119886119889119905)
[119886]2+ [119887]2
(A3)
Computational Intelligence and Neuroscience 19
Now
119889119886
119889119905
(120596119909 120596119910 119905)
= int
R2
120597119906
120597119905
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
(A4)
For a purely translational signal with instantaneous spatialshift given by (119904
119909(119905) 119904119910(119905)) within the support of the window
we have
119889119886
119889119905
(120596119909 120596119910 119905)
= minusint
R2
119889119904119909
119889119905
(119905)
120597119906
120597119909
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus int
R2
119889119904119910
119889119905
(119905)
120597119906
120597119910
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119909
119889119904119909
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119910
119889119904119910
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199033=1199033(120596119909120596119910119905)
+ 120596119909
119889119904119909
119889119905
119887
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199034=1199034(120596119909120596119910119905)
+ 120596119910
119889119904119910
119889119905
119887
(A5)
Similarly
119889119887
119889119905
(120596119909 120596119910 119905) = minus
119889119904119909
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199035=1199035(120596119909120596119910119905)
minus 120596119909
119889119904119909
119889119905
119886 minus
119889119904119910
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199036=1199036(120596119909120596119910119905)
minus 120596119910
119889119904119910
119889119905
119886
(A6)
Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to
11988912060100
119889119905
(120596119909 120596119910 119905) =
minus (119889119904119909119889119905) [119903
3119887 + 1199035119886 + 120596119909 [
119886]2+ 120596119909 [
119887]2] minus (119889119904
119910119889119905) [119903
4119887 + 1199036119886 + 120596119910 [
119886]2+ 120596119910 [
119887]2]
[119886]2+ [119887]2
= minus
119889119904119909
119889119905
120596119909minus
119889119904119910
119889119905
120596119910minus
119889119904119909
119889119905
sdot
1199033119887 + 1199035119886
[119886]2+ [119887]2minus
119889119904119910
119889119905
sdot
1199034119887 + 1199036119886
[119886]2+ [119887]2
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
v00(120596119909120596119910119905)
(A7)
Disclosure
The authorrsquos names are alphabetically listed
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgment
The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232
References
[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010
[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010
[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014
[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010
20 Computational Intelligence and Neuroscience
[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013
[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014
[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956
[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010
[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985
[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965
[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981
[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015
[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014
[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999
[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015
[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001
[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996
[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980
[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013
[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013
[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014
[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014
[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach
to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012
[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990
[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987
[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998
Submit your manuscripts athttpwwwhindawicom
Computer Games Technology
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Distributed Sensor Networks
International Journal of
Advances in
FuzzySystems
Hindawi Publishing Corporationhttpwwwhindawicom
Volume 2014
International Journal of
ReconfigurableComputing
Hindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Applied Computational Intelligence and Soft Computing
thinspAdvancesthinspinthinsp
Artificial Intelligence
HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014
Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Journal of
Computer Networks and Communications
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation
httpwwwhindawicom Volume 2014
Advances in
Multimedia
International Journal of
Biomedical Imaging
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
ArtificialNeural Systems
Advances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience
Industrial EngineeringJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Human-ComputerInteraction
Advances in
Computer EngineeringAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
12 Computational Intelligence and Neuroscience
ukl(t) uk+1l(t)
g1(t)
g2(t)
g1(t)
g2(t)
+ minus+
timestimes
(a)
minus
minus minus
+
ukl(t) uk+1l(t)
g1(t)g1(t)
g2(t) g2(t)
++
+
(b)
Figure 8 (a) A Reichardt motion detector detecting motion between the two points 119906119896119897and 119906
119896+1119897in the blurred and downsampled video
(b) A Barlow-Levick motion detector detecting motion between the two points 119906119896119897and 119906
119896+1119897in the blurred and downsampled video 119892
1(119905)
denotes a high-pass filter and 1198922(119905) denotes a low-pass filter
moved the high-pass filter ℎ1(119905) to the front of the low-pass fil-
ter ℎ2(119905) This configuration provides a superior performance
to the one in Figure 7 For both the Reichardt and the Barlow-Levick detectors the videos are first blurred by a Gaussianfilter (with the same variance as the Gaussian window in (30)used for the phase-based motion detector) and subsampledat the center of each overlapping block in the phase-basedmotion detector The subsampled video then provides inputsto two 2D arrays of the circuits shown in Figure 8 one forthe horizontal direction and one for the vertical directionThe outputs of the horizontal and vertical motion circuitsform an array of motion vectors that indicate the strengthand direction of motion The three tested motion detectorshave the same number of outputs as a result The processingspeeds of the Reichardt motion detector and the Barlow-Levick motion detector both implemented in PyCUDA andtested on the same GPU are shown in Table 1
Note that the Reichardt motion detector and the Barlow-Levick motion detector are highly efficient due to the sim-plicity of their algorithmsThe phase-basedmotion detectionalgorithm however is a much more sophisticated algorithmand yet it can be implemented in real-time using parallelcomputing devices
The fast GPU implementation is based on the FFTand Matrix-Matrix multiplication It is expected that thoseoperations can be efficiently implemented in hardware forexample FPGA
42 Examples of Phased-BasedMotionDetection Weappliedour motion detection algorithm on video sequences of theChange Detection 2014 dataset [22] that did not exhibitcamera egomotion For these video sequences the standarddeviation of the Gaussian window functions was set to 4
pixels and the block size was chosen to be 32 times 32 pixelsThreshold and normalization parameters were kept the samewith the exception of the ldquothermal videordquo (in order to dealwith larger background noise levels see below) We alsotested the same video sequences using the Reichardt motiondetector and the Barlow-Levick motion detector For theReichardt detector the high-pass filters were chosen as firstorder filters with a time constant of 200 milliseconds and thelow-pass filters were chosen as first order filters with a timeconstant of 300 milliseconds (assuming that the frame rate is50 frames per second)Thresholdwas set to 2 For the Barlow-Levick motion detector the time constant of the first high-pass filters was set to 250 milliseconds The low-pass filterswere the same as in the Reichardt detectorThreshold was setto 2
The first video was taken from a highway surveillancecamera (ldquohighway videordquo) under good illumination con-ditions and high contrast The video had moderate noiseparticularly on the road surface The detected motion isshown in the top left panel of Figure 9 (see SupplementaryVideo S2 for full video) The phase-based motion detectionalgorithm captured both the moving cars and the tree leavesmoving (due to the wind) In the time interval between the9th and 10th second the camera was slightly moved left-and right-wards within 5 frames again possibly due to thewind Movement due to this shift was captured by themotiondetection algorithm and the algorithm was fast enough tocorrectly determine the direction of this movement In thisvideo we already noted that this algorithm suffers from theaperture problem For example in front of the van where along horizontal edge is present the detectedmotion ismostlypointing downwards In addition to moving downwards theedge is alsomoving to the left howeverThis is expected since
Computational Intelligence and Neuroscience 13
(a) (b) (c)
Figure 9 Top motion detection of the ldquohighway videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) Red arrows indicate detected motion and its direction Bottom the contrast of the video was artificiallyreduced by 5-fold and the mean was reduced to 35 of the original The red arrows are duplicated from the motion detection result on theoriginal video as in top Blue arrows are the result of motion detection on the video with reduced contrast If motion is detected both in theoriginal video and in the video with reduced contrast then the arrow is shown in magenta (as a mix of blue and red) (see SupplementaryVideo S2 for full video)
the algorithm only detects motion locally and does not takeinto account the overall shape of any object
For comparison the motion detection results for theReichardt motion detector and the Barlow-Levick motiondetector are shown in the top middle and top right panelof Figure 9 (see Supplementary Video S2 for full video) TheReichardt detector performed relatively well when vehiclesare moving faster but the direction it predicts for vehiclesmoving slower for example on the back of the image is notaccurate In addition motion is still detected in some partsof the screen where the vehicles have just passed by Thedetection result for the Barlow-Levick motion detector waspoorer In particular the response to OFF edge movement isalways the opposite to the actual movement direction
We then squeezed the range of the screen intensity from[0 1] to [02 04] resulting in a video with a lower meanluminance and lower contrast The motion detection resultson the low-contrast video are shown in the bottom 3 panelsof Figure 9 (see Supplementary Video S2 for full video) Forreference the motion detection results for the original videoare shown in red arrows Motion detected in the low-contrastvideo is shown in blue arrows if no motion is detected forthe block in the original video If motion is detected in boththe original and the low-contrast video the arrows are shownin magenta Figure 9 clearly shows that while the motiondetection performance is degraded for all three detectors thephase-basedmotion detector performed still quite well in thelow-contrast video detecting most of the moving vehicles
The other two detectors missed many of the blocks wheremotion was detected in the original movie
To quantify how well each detector works under differentcontrast conditions we computed the ratio of unthresholdedoutput values of eachmotion detector between lower contrastand full contrast video For example in the case of phase-basedmotion detector we computed for each block the ratiobetween the PMI index for the lower contrast video and thatfor the full contrast videoThe ratios are then averaged acrossall blocks where motion is detected in the full contrast videoThis average for different contrasts is shown in Figure 10 bythe blue curve In the ideal case when the normalizationconstant 120598 is 0 the phase detector should produce invariantresponses to videos with different contrast The curve shownhere is mainly due to a nonzero 120598The ratios for the Reichardtmotion detector and the Barlow-Levick motion detectorare computed similarly and are shown in red and yellowrespectively It is clear that the phase-based motion detectorhas a superior performance across a range of contrast valuesAt 20 contrast the phase-based detector still has 50 ofthe PMI index value for full contrast video As expectedthe response of Barlow-Levick motion detector is linear withrespect to contrast and the Reichardt motion detector has aquadratic relation to contrast For a fixed threshold value alarger ratio equates to amore consistent performance at lowercontrast
The second video was captured by a surveillance camerain a train station (ldquotrain station videordquo) The video was
14 Computational Intelligence and Neuroscience
Phase-basedReichardtBarlow-Levick
0
02
04
06
08
1
Ratio
40 60 80200 100Contrast ()
Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined
under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm
The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)
The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression
With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)
43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient
objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area
For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold
We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually
We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm
We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)
These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm
5 Discussion
Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes
Computational Intelligence and Neuroscience 15
(a) (b) (c)
Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)
(a) (b) (c)
Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)
by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to
models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]
We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in
16 Computational Intelligence and Neuroscience
(a) (b) (c)
Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)
Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)
phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in
Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)
modeling in vivo biological vision systems This is consistentwith other recent findings [15]
Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure
Computational Intelligence and Neuroscience 17
(a) (b)
Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)
(a) (b)
Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)
(a) (b)
Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)
Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase
itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third
18 Computational Intelligence and Neuroscience
(a) (b)
Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)
instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection
The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions
Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors
We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well
We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It
can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes
Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems
Appendix
The Derivative of the Local Phase
Let
119886 (120596119909 120596119910 119905)
= int
R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596
119909119909 + 120596119910119910) 119889119909 119889119910
119887 (120596119909 120596119910 119905)
= minusint
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
(A1)
and therefore
11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)
= 119886 (120596119909 120596119910 119905) + 119895119887 (120596
119909 120596119910 119905)
(A2)
The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596
119909
120596119910 119905) 119886(120596
119909 120596119910 119905)) and therefore
11988912060100
119889119905
=
119886 (119889119887119889119905) minus 119887 (119889119886119889119905)
[119886]2+ [119887]2
(A3)
Computational Intelligence and Neuroscience 19
Now
119889119886
119889119905
(120596119909 120596119910 119905)
= int
R2
120597119906
120597119905
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
(A4)
For a purely translational signal with instantaneous spatialshift given by (119904
119909(119905) 119904119910(119905)) within the support of the window
we have
119889119886
119889119905
(120596119909 120596119910 119905)
= minusint
R2
119889119904119909
119889119905
(119905)
120597119906
120597119909
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus int
R2
119889119904119910
119889119905
(119905)
120597119906
120597119910
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119909
119889119904119909
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119910
119889119904119910
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199033=1199033(120596119909120596119910119905)
+ 120596119909
119889119904119909
119889119905
119887
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199034=1199034(120596119909120596119910119905)
+ 120596119910
119889119904119910
119889119905
119887
(A5)
Similarly
119889119887
119889119905
(120596119909 120596119910 119905) = minus
119889119904119909
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199035=1199035(120596119909120596119910119905)
minus 120596119909
119889119904119909
119889119905
119886 minus
119889119904119910
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199036=1199036(120596119909120596119910119905)
minus 120596119910
119889119904119910
119889119905
119886
(A6)
Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to
11988912060100
119889119905
(120596119909 120596119910 119905) =
minus (119889119904119909119889119905) [119903
3119887 + 1199035119886 + 120596119909 [
119886]2+ 120596119909 [
119887]2] minus (119889119904
119910119889119905) [119903
4119887 + 1199036119886 + 120596119910 [
119886]2+ 120596119910 [
119887]2]
[119886]2+ [119887]2
= minus
119889119904119909
119889119905
120596119909minus
119889119904119910
119889119905
120596119910minus
119889119904119909
119889119905
sdot
1199033119887 + 1199035119886
[119886]2+ [119887]2minus
119889119904119910
119889119905
sdot
1199034119887 + 1199036119886
[119886]2+ [119887]2
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
v00(120596119909120596119910119905)
(A7)
Disclosure
The authorrsquos names are alphabetically listed
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgment
The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232
References
[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010
[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010
[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014
[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010
20 Computational Intelligence and Neuroscience
[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013
[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014
[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956
[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010
[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985
[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965
[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981
[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015
[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014
[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999
[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015
[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001
[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996
[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980
[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013
[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013
[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014
[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014
[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach
to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012
[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990
[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987
[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998
Submit your manuscripts athttpwwwhindawicom
Computer Games Technology
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Distributed Sensor Networks
International Journal of
Advances in
FuzzySystems
Hindawi Publishing Corporationhttpwwwhindawicom
Volume 2014
International Journal of
ReconfigurableComputing
Hindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Applied Computational Intelligence and Soft Computing
thinspAdvancesthinspinthinsp
Artificial Intelligence
HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014
Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Journal of
Computer Networks and Communications
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation
httpwwwhindawicom Volume 2014
Advances in
Multimedia
International Journal of
Biomedical Imaging
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
ArtificialNeural Systems
Advances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience
Industrial EngineeringJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Human-ComputerInteraction
Advances in
Computer EngineeringAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience 13
(a) (b) (c)
Figure 9 Top motion detection of the ldquohighway videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) Red arrows indicate detected motion and its direction Bottom the contrast of the video was artificiallyreduced by 5-fold and the mean was reduced to 35 of the original The red arrows are duplicated from the motion detection result on theoriginal video as in top Blue arrows are the result of motion detection on the video with reduced contrast If motion is detected both in theoriginal video and in the video with reduced contrast then the arrow is shown in magenta (as a mix of blue and red) (see SupplementaryVideo S2 for full video)
the algorithm only detects motion locally and does not takeinto account the overall shape of any object
For comparison the motion detection results for theReichardt motion detector and the Barlow-Levick motiondetector are shown in the top middle and top right panelof Figure 9 (see Supplementary Video S2 for full video) TheReichardt detector performed relatively well when vehiclesare moving faster but the direction it predicts for vehiclesmoving slower for example on the back of the image is notaccurate In addition motion is still detected in some partsof the screen where the vehicles have just passed by Thedetection result for the Barlow-Levick motion detector waspoorer In particular the response to OFF edge movement isalways the opposite to the actual movement direction
We then squeezed the range of the screen intensity from[0 1] to [02 04] resulting in a video with a lower meanluminance and lower contrast The motion detection resultson the low-contrast video are shown in the bottom 3 panelsof Figure 9 (see Supplementary Video S2 for full video) Forreference the motion detection results for the original videoare shown in red arrows Motion detected in the low-contrastvideo is shown in blue arrows if no motion is detected forthe block in the original video If motion is detected in boththe original and the low-contrast video the arrows are shownin magenta Figure 9 clearly shows that while the motiondetection performance is degraded for all three detectors thephase-basedmotion detector performed still quite well in thelow-contrast video detecting most of the moving vehicles
The other two detectors missed many of the blocks wheremotion was detected in the original movie
To quantify how well each detector works under differentcontrast conditions we computed the ratio of unthresholdedoutput values of eachmotion detector between lower contrastand full contrast video For example in the case of phase-basedmotion detector we computed for each block the ratiobetween the PMI index for the lower contrast video and thatfor the full contrast videoThe ratios are then averaged acrossall blocks where motion is detected in the full contrast videoThis average for different contrasts is shown in Figure 10 bythe blue curve In the ideal case when the normalizationconstant 120598 is 0 the phase detector should produce invariantresponses to videos with different contrast The curve shownhere is mainly due to a nonzero 120598The ratios for the Reichardtmotion detector and the Barlow-Levick motion detectorare computed similarly and are shown in red and yellowrespectively It is clear that the phase-based motion detectorhas a superior performance across a range of contrast valuesAt 20 contrast the phase-based detector still has 50 ofthe PMI index value for full contrast video As expectedthe response of Barlow-Levick motion detector is linear withrespect to contrast and the Reichardt motion detector has aquadratic relation to contrast For a fixed threshold value alarger ratio equates to amore consistent performance at lowercontrast
The second video was captured by a surveillance camerain a train station (ldquotrain station videordquo) The video was
14 Computational Intelligence and Neuroscience
Phase-basedReichardtBarlow-Levick
0
02
04
06
08
1
Ratio
40 60 80200 100Contrast ()
Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined
under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm
The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)
The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression
With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)
43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient
objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area
For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold
We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually
We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm
We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)
These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm
5 Discussion
Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes
Computational Intelligence and Neuroscience 15
(a) (b) (c)
Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)
(a) (b) (c)
Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)
by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to
models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]
We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in
16 Computational Intelligence and Neuroscience
(a) (b) (c)
Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)
Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)
phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in
Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)
modeling in vivo biological vision systems This is consistentwith other recent findings [15]
Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure
Computational Intelligence and Neuroscience 17
(a) (b)
Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)
(a) (b)
Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)
(a) (b)
Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)
Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase
itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third
18 Computational Intelligence and Neuroscience
(a) (b)
Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)
instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection
The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions
Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors
We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well
We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It
can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes
Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems
Appendix
The Derivative of the Local Phase
Let
119886 (120596119909 120596119910 119905)
= int
R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596
119909119909 + 120596119910119910) 119889119909 119889119910
119887 (120596119909 120596119910 119905)
= minusint
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
(A1)
and therefore
11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)
= 119886 (120596119909 120596119910 119905) + 119895119887 (120596
119909 120596119910 119905)
(A2)
The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596
119909
120596119910 119905) 119886(120596
119909 120596119910 119905)) and therefore
11988912060100
119889119905
=
119886 (119889119887119889119905) minus 119887 (119889119886119889119905)
[119886]2+ [119887]2
(A3)
Computational Intelligence and Neuroscience 19
Now
119889119886
119889119905
(120596119909 120596119910 119905)
= int
R2
120597119906
120597119905
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
(A4)
For a purely translational signal with instantaneous spatialshift given by (119904
119909(119905) 119904119910(119905)) within the support of the window
we have
119889119886
119889119905
(120596119909 120596119910 119905)
= minusint
R2
119889119904119909
119889119905
(119905)
120597119906
120597119909
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus int
R2
119889119904119910
119889119905
(119905)
120597119906
120597119910
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119909
119889119904119909
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119910
119889119904119910
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199033=1199033(120596119909120596119910119905)
+ 120596119909
119889119904119909
119889119905
119887
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199034=1199034(120596119909120596119910119905)
+ 120596119910
119889119904119910
119889119905
119887
(A5)
Similarly
119889119887
119889119905
(120596119909 120596119910 119905) = minus
119889119904119909
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199035=1199035(120596119909120596119910119905)
minus 120596119909
119889119904119909
119889119905
119886 minus
119889119904119910
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199036=1199036(120596119909120596119910119905)
minus 120596119910
119889119904119910
119889119905
119886
(A6)
Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to
11988912060100
119889119905
(120596119909 120596119910 119905) =
minus (119889119904119909119889119905) [119903
3119887 + 1199035119886 + 120596119909 [
119886]2+ 120596119909 [
119887]2] minus (119889119904
119910119889119905) [119903
4119887 + 1199036119886 + 120596119910 [
119886]2+ 120596119910 [
119887]2]
[119886]2+ [119887]2
= minus
119889119904119909
119889119905
120596119909minus
119889119904119910
119889119905
120596119910minus
119889119904119909
119889119905
sdot
1199033119887 + 1199035119886
[119886]2+ [119887]2minus
119889119904119910
119889119905
sdot
1199034119887 + 1199036119886
[119886]2+ [119887]2
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
v00(120596119909120596119910119905)
(A7)
Disclosure
The authorrsquos names are alphabetically listed
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgment
The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232
References
[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010
[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010
[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014
[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010
20 Computational Intelligence and Neuroscience
[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013
[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014
[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956
[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010
[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985
[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965
[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981
[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015
[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014
[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999
[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015
[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001
[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996
[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980
[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013
[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013
[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014
[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014
[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach
to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012
[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990
[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987
[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998
Submit your manuscripts athttpwwwhindawicom
Computer Games Technology
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Distributed Sensor Networks
International Journal of
Advances in
FuzzySystems
Hindawi Publishing Corporationhttpwwwhindawicom
Volume 2014
International Journal of
ReconfigurableComputing
Hindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Applied Computational Intelligence and Soft Computing
thinspAdvancesthinspinthinsp
Artificial Intelligence
HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014
Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Journal of
Computer Networks and Communications
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation
httpwwwhindawicom Volume 2014
Advances in
Multimedia
International Journal of
Biomedical Imaging
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
ArtificialNeural Systems
Advances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience
Industrial EngineeringJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Human-ComputerInteraction
Advances in
Computer EngineeringAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
14 Computational Intelligence and Neuroscience
Phase-basedReichardtBarlow-Levick
0
02
04
06
08
1
Ratio
40 60 80200 100Contrast ()
Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined
under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm
The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)
The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression
With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)
43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient
objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area
For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold
We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually
We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm
We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)
These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm
5 Discussion
Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes
Computational Intelligence and Neuroscience 15
(a) (b) (c)
Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)
(a) (b) (c)
Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)
by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to
models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]
We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in
16 Computational Intelligence and Neuroscience
(a) (b) (c)
Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)
Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)
phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in
Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)
modeling in vivo biological vision systems This is consistentwith other recent findings [15]
Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure
Computational Intelligence and Neuroscience 17
(a) (b)
Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)
(a) (b)
Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)
(a) (b)
Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)
Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase
itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third
18 Computational Intelligence and Neuroscience
(a) (b)
Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)
instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection
The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions
Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors
We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well
We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It
can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes
Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems
Appendix
The Derivative of the Local Phase
Let
119886 (120596119909 120596119910 119905)
= int
R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596
119909119909 + 120596119910119910) 119889119909 119889119910
119887 (120596119909 120596119910 119905)
= minusint
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
(A1)
and therefore
11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)
= 119886 (120596119909 120596119910 119905) + 119895119887 (120596
119909 120596119910 119905)
(A2)
The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596
119909
120596119910 119905) 119886(120596
119909 120596119910 119905)) and therefore
11988912060100
119889119905
=
119886 (119889119887119889119905) minus 119887 (119889119886119889119905)
[119886]2+ [119887]2
(A3)
Computational Intelligence and Neuroscience 19
Now
119889119886
119889119905
(120596119909 120596119910 119905)
= int
R2
120597119906
120597119905
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
(A4)
For a purely translational signal with instantaneous spatialshift given by (119904
119909(119905) 119904119910(119905)) within the support of the window
we have
119889119886
119889119905
(120596119909 120596119910 119905)
= minusint
R2
119889119904119909
119889119905
(119905)
120597119906
120597119909
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus int
R2
119889119904119910
119889119905
(119905)
120597119906
120597119910
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119909
119889119904119909
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119910
119889119904119910
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199033=1199033(120596119909120596119910119905)
+ 120596119909
119889119904119909
119889119905
119887
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199034=1199034(120596119909120596119910119905)
+ 120596119910
119889119904119910
119889119905
119887
(A5)
Similarly
119889119887
119889119905
(120596119909 120596119910 119905) = minus
119889119904119909
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199035=1199035(120596119909120596119910119905)
minus 120596119909
119889119904119909
119889119905
119886 minus
119889119904119910
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199036=1199036(120596119909120596119910119905)
minus 120596119910
119889119904119910
119889119905
119886
(A6)
Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to
11988912060100
119889119905
(120596119909 120596119910 119905) =
minus (119889119904119909119889119905) [119903
3119887 + 1199035119886 + 120596119909 [
119886]2+ 120596119909 [
119887]2] minus (119889119904
119910119889119905) [119903
4119887 + 1199036119886 + 120596119910 [
119886]2+ 120596119910 [
119887]2]
[119886]2+ [119887]2
= minus
119889119904119909
119889119905
120596119909minus
119889119904119910
119889119905
120596119910minus
119889119904119909
119889119905
sdot
1199033119887 + 1199035119886
[119886]2+ [119887]2minus
119889119904119910
119889119905
sdot
1199034119887 + 1199036119886
[119886]2+ [119887]2
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
v00(120596119909120596119910119905)
(A7)
Disclosure
The authorrsquos names are alphabetically listed
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgment
The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232
References
[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010
[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010
[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014
[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010
20 Computational Intelligence and Neuroscience
[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013
[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014
[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956
[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010
[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985
[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965
[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981
[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015
[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014
[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999
[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015
[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001
[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996
[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980
[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013
[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013
[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014
[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014
[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach
to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012
[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990
[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987
[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998
Submit your manuscripts athttpwwwhindawicom
Computer Games Technology
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Distributed Sensor Networks
International Journal of
Advances in
FuzzySystems
Hindawi Publishing Corporationhttpwwwhindawicom
Volume 2014
International Journal of
ReconfigurableComputing
Hindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Applied Computational Intelligence and Soft Computing
thinspAdvancesthinspinthinsp
Artificial Intelligence
HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014
Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Journal of
Computer Networks and Communications
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation
httpwwwhindawicom Volume 2014
Advances in
Multimedia
International Journal of
Biomedical Imaging
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
ArtificialNeural Systems
Advances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience
Industrial EngineeringJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Human-ComputerInteraction
Advances in
Computer EngineeringAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience 15
(a) (b) (c)
Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)
(a) (b) (c)
Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)
by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to
models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]
We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in
16 Computational Intelligence and Neuroscience
(a) (b) (c)
Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)
Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)
phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in
Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)
modeling in vivo biological vision systems This is consistentwith other recent findings [15]
Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure
Computational Intelligence and Neuroscience 17
(a) (b)
Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)
(a) (b)
Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)
(a) (b)
Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)
Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase
itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third
18 Computational Intelligence and Neuroscience
(a) (b)
Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)
instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection
The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions
Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors
We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well
We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It
can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes
Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems
Appendix
The Derivative of the Local Phase
Let
119886 (120596119909 120596119910 119905)
= int
R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596
119909119909 + 120596119910119910) 119889119909 119889119910
119887 (120596119909 120596119910 119905)
= minusint
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
(A1)
and therefore
11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)
= 119886 (120596119909 120596119910 119905) + 119895119887 (120596
119909 120596119910 119905)
(A2)
The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596
119909
120596119910 119905) 119886(120596
119909 120596119910 119905)) and therefore
11988912060100
119889119905
=
119886 (119889119887119889119905) minus 119887 (119889119886119889119905)
[119886]2+ [119887]2
(A3)
Computational Intelligence and Neuroscience 19
Now
119889119886
119889119905
(120596119909 120596119910 119905)
= int
R2
120597119906
120597119905
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
(A4)
For a purely translational signal with instantaneous spatialshift given by (119904
119909(119905) 119904119910(119905)) within the support of the window
we have
119889119886
119889119905
(120596119909 120596119910 119905)
= minusint
R2
119889119904119909
119889119905
(119905)
120597119906
120597119909
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus int
R2
119889119904119910
119889119905
(119905)
120597119906
120597119910
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119909
119889119904119909
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119910
119889119904119910
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199033=1199033(120596119909120596119910119905)
+ 120596119909
119889119904119909
119889119905
119887
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199034=1199034(120596119909120596119910119905)
+ 120596119910
119889119904119910
119889119905
119887
(A5)
Similarly
119889119887
119889119905
(120596119909 120596119910 119905) = minus
119889119904119909
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199035=1199035(120596119909120596119910119905)
minus 120596119909
119889119904119909
119889119905
119886 minus
119889119904119910
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199036=1199036(120596119909120596119910119905)
minus 120596119910
119889119904119910
119889119905
119886
(A6)
Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to
11988912060100
119889119905
(120596119909 120596119910 119905) =
minus (119889119904119909119889119905) [119903
3119887 + 1199035119886 + 120596119909 [
119886]2+ 120596119909 [
119887]2] minus (119889119904
119910119889119905) [119903
4119887 + 1199036119886 + 120596119910 [
119886]2+ 120596119910 [
119887]2]
[119886]2+ [119887]2
= minus
119889119904119909
119889119905
120596119909minus
119889119904119910
119889119905
120596119910minus
119889119904119909
119889119905
sdot
1199033119887 + 1199035119886
[119886]2+ [119887]2minus
119889119904119910
119889119905
sdot
1199034119887 + 1199036119886
[119886]2+ [119887]2
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
v00(120596119909120596119910119905)
(A7)
Disclosure
The authorrsquos names are alphabetically listed
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgment
The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232
References
[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010
[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010
[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014
[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010
20 Computational Intelligence and Neuroscience
[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013
[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014
[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956
[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010
[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985
[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965
[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981
[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015
[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014
[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999
[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015
[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001
[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996
[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980
[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013
[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013
[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014
[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014
[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach
to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012
[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990
[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987
[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998
Submit your manuscripts athttpwwwhindawicom
Computer Games Technology
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Distributed Sensor Networks
International Journal of
Advances in
FuzzySystems
Hindawi Publishing Corporationhttpwwwhindawicom
Volume 2014
International Journal of
ReconfigurableComputing
Hindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Applied Computational Intelligence and Soft Computing
thinspAdvancesthinspinthinsp
Artificial Intelligence
HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014
Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Journal of
Computer Networks and Communications
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation
httpwwwhindawicom Volume 2014
Advances in
Multimedia
International Journal of
Biomedical Imaging
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
ArtificialNeural Systems
Advances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience
Industrial EngineeringJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Human-ComputerInteraction
Advances in
Computer EngineeringAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
16 Computational Intelligence and Neuroscience
(a) (b) (c)
Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)
Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)
phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in
Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)
modeling in vivo biological vision systems This is consistentwith other recent findings [15]
Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure
Computational Intelligence and Neuroscience 17
(a) (b)
Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)
(a) (b)
Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)
(a) (b)
Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)
Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase
itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third
18 Computational Intelligence and Neuroscience
(a) (b)
Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)
instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection
The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions
Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors
We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well
We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It
can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes
Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems
Appendix
The Derivative of the Local Phase
Let
119886 (120596119909 120596119910 119905)
= int
R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596
119909119909 + 120596119910119910) 119889119909 119889119910
119887 (120596119909 120596119910 119905)
= minusint
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
(A1)
and therefore
11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)
= 119886 (120596119909 120596119910 119905) + 119895119887 (120596
119909 120596119910 119905)
(A2)
The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596
119909
120596119910 119905) 119886(120596
119909 120596119910 119905)) and therefore
11988912060100
119889119905
=
119886 (119889119887119889119905) minus 119887 (119889119886119889119905)
[119886]2+ [119887]2
(A3)
Computational Intelligence and Neuroscience 19
Now
119889119886
119889119905
(120596119909 120596119910 119905)
= int
R2
120597119906
120597119905
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
(A4)
For a purely translational signal with instantaneous spatialshift given by (119904
119909(119905) 119904119910(119905)) within the support of the window
we have
119889119886
119889119905
(120596119909 120596119910 119905)
= minusint
R2
119889119904119909
119889119905
(119905)
120597119906
120597119909
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus int
R2
119889119904119910
119889119905
(119905)
120597119906
120597119910
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119909
119889119904119909
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119910
119889119904119910
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199033=1199033(120596119909120596119910119905)
+ 120596119909
119889119904119909
119889119905
119887
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199034=1199034(120596119909120596119910119905)
+ 120596119910
119889119904119910
119889119905
119887
(A5)
Similarly
119889119887
119889119905
(120596119909 120596119910 119905) = minus
119889119904119909
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199035=1199035(120596119909120596119910119905)
minus 120596119909
119889119904119909
119889119905
119886 minus
119889119904119910
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199036=1199036(120596119909120596119910119905)
minus 120596119910
119889119904119910
119889119905
119886
(A6)
Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to
11988912060100
119889119905
(120596119909 120596119910 119905) =
minus (119889119904119909119889119905) [119903
3119887 + 1199035119886 + 120596119909 [
119886]2+ 120596119909 [
119887]2] minus (119889119904
119910119889119905) [119903
4119887 + 1199036119886 + 120596119910 [
119886]2+ 120596119910 [
119887]2]
[119886]2+ [119887]2
= minus
119889119904119909
119889119905
120596119909minus
119889119904119910
119889119905
120596119910minus
119889119904119909
119889119905
sdot
1199033119887 + 1199035119886
[119886]2+ [119887]2minus
119889119904119910
119889119905
sdot
1199034119887 + 1199036119886
[119886]2+ [119887]2
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
v00(120596119909120596119910119905)
(A7)
Disclosure
The authorrsquos names are alphabetically listed
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgment
The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232
References
[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010
[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010
[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014
[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010
20 Computational Intelligence and Neuroscience
[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013
[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014
[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956
[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010
[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985
[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965
[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981
[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015
[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014
[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999
[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015
[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001
[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996
[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980
[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013
[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013
[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014
[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014
[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach
to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012
[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990
[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987
[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998
Submit your manuscripts athttpwwwhindawicom
Computer Games Technology
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Distributed Sensor Networks
International Journal of
Advances in
FuzzySystems
Hindawi Publishing Corporationhttpwwwhindawicom
Volume 2014
International Journal of
ReconfigurableComputing
Hindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Applied Computational Intelligence and Soft Computing
thinspAdvancesthinspinthinsp
Artificial Intelligence
HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014
Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Journal of
Computer Networks and Communications
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation
httpwwwhindawicom Volume 2014
Advances in
Multimedia
International Journal of
Biomedical Imaging
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
ArtificialNeural Systems
Advances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience
Industrial EngineeringJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Human-ComputerInteraction
Advances in
Computer EngineeringAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience 17
(a) (b)
Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)
(a) (b)
Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)
(a) (b)
Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)
Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase
itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third
18 Computational Intelligence and Neuroscience
(a) (b)
Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)
instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection
The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions
Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors
We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well
We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It
can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes
Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems
Appendix
The Derivative of the Local Phase
Let
119886 (120596119909 120596119910 119905)
= int
R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596
119909119909 + 120596119910119910) 119889119909 119889119910
119887 (120596119909 120596119910 119905)
= minusint
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
(A1)
and therefore
11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)
= 119886 (120596119909 120596119910 119905) + 119895119887 (120596
119909 120596119910 119905)
(A2)
The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596
119909
120596119910 119905) 119886(120596
119909 120596119910 119905)) and therefore
11988912060100
119889119905
=
119886 (119889119887119889119905) minus 119887 (119889119886119889119905)
[119886]2+ [119887]2
(A3)
Computational Intelligence and Neuroscience 19
Now
119889119886
119889119905
(120596119909 120596119910 119905)
= int
R2
120597119906
120597119905
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
(A4)
For a purely translational signal with instantaneous spatialshift given by (119904
119909(119905) 119904119910(119905)) within the support of the window
we have
119889119886
119889119905
(120596119909 120596119910 119905)
= minusint
R2
119889119904119909
119889119905
(119905)
120597119906
120597119909
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus int
R2
119889119904119910
119889119905
(119905)
120597119906
120597119910
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119909
119889119904119909
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119910
119889119904119910
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199033=1199033(120596119909120596119910119905)
+ 120596119909
119889119904119909
119889119905
119887
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199034=1199034(120596119909120596119910119905)
+ 120596119910
119889119904119910
119889119905
119887
(A5)
Similarly
119889119887
119889119905
(120596119909 120596119910 119905) = minus
119889119904119909
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199035=1199035(120596119909120596119910119905)
minus 120596119909
119889119904119909
119889119905
119886 minus
119889119904119910
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199036=1199036(120596119909120596119910119905)
minus 120596119910
119889119904119910
119889119905
119886
(A6)
Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to
11988912060100
119889119905
(120596119909 120596119910 119905) =
minus (119889119904119909119889119905) [119903
3119887 + 1199035119886 + 120596119909 [
119886]2+ 120596119909 [
119887]2] minus (119889119904
119910119889119905) [119903
4119887 + 1199036119886 + 120596119910 [
119886]2+ 120596119910 [
119887]2]
[119886]2+ [119887]2
= minus
119889119904119909
119889119905
120596119909minus
119889119904119910
119889119905
120596119910minus
119889119904119909
119889119905
sdot
1199033119887 + 1199035119886
[119886]2+ [119887]2minus
119889119904119910
119889119905
sdot
1199034119887 + 1199036119886
[119886]2+ [119887]2
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
v00(120596119909120596119910119905)
(A7)
Disclosure
The authorrsquos names are alphabetically listed
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgment
The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232
References
[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010
[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010
[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014
[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010
20 Computational Intelligence and Neuroscience
[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013
[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014
[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956
[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010
[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985
[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965
[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981
[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015
[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014
[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999
[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015
[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001
[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996
[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980
[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013
[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013
[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014
[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014
[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach
to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012
[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990
[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987
[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998
Submit your manuscripts athttpwwwhindawicom
Computer Games Technology
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Distributed Sensor Networks
International Journal of
Advances in
FuzzySystems
Hindawi Publishing Corporationhttpwwwhindawicom
Volume 2014
International Journal of
ReconfigurableComputing
Hindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Applied Computational Intelligence and Soft Computing
thinspAdvancesthinspinthinsp
Artificial Intelligence
HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014
Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Journal of
Computer Networks and Communications
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation
httpwwwhindawicom Volume 2014
Advances in
Multimedia
International Journal of
Biomedical Imaging
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
ArtificialNeural Systems
Advances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience
Industrial EngineeringJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Human-ComputerInteraction
Advances in
Computer EngineeringAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
18 Computational Intelligence and Neuroscience
(a) (b)
Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)
instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection
The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions
Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors
We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well
We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It
can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes
Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems
Appendix
The Derivative of the Local Phase
Let
119886 (120596119909 120596119910 119905)
= int
R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596
119909119909 + 120596119910119910) 119889119909 119889119910
119887 (120596119909 120596119910 119905)
= minusint
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
(A1)
and therefore
11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)
= 119886 (120596119909 120596119910 119905) + 119895119887 (120596
119909 120596119910 119905)
(A2)
The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596
119909
120596119910 119905) 119886(120596
119909 120596119910 119905)) and therefore
11988912060100
119889119905
=
119886 (119889119887119889119905) minus 119887 (119889119886119889119905)
[119886]2+ [119887]2
(A3)
Computational Intelligence and Neuroscience 19
Now
119889119886
119889119905
(120596119909 120596119910 119905)
= int
R2
120597119906
120597119905
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
(A4)
For a purely translational signal with instantaneous spatialshift given by (119904
119909(119905) 119904119910(119905)) within the support of the window
we have
119889119886
119889119905
(120596119909 120596119910 119905)
= minusint
R2
119889119904119909
119889119905
(119905)
120597119906
120597119909
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus int
R2
119889119904119910
119889119905
(119905)
120597119906
120597119910
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119909
119889119904119909
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119910
119889119904119910
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199033=1199033(120596119909120596119910119905)
+ 120596119909
119889119904119909
119889119905
119887
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199034=1199034(120596119909120596119910119905)
+ 120596119910
119889119904119910
119889119905
119887
(A5)
Similarly
119889119887
119889119905
(120596119909 120596119910 119905) = minus
119889119904119909
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199035=1199035(120596119909120596119910119905)
minus 120596119909
119889119904119909
119889119905
119886 minus
119889119904119910
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199036=1199036(120596119909120596119910119905)
minus 120596119910
119889119904119910
119889119905
119886
(A6)
Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to
11988912060100
119889119905
(120596119909 120596119910 119905) =
minus (119889119904119909119889119905) [119903
3119887 + 1199035119886 + 120596119909 [
119886]2+ 120596119909 [
119887]2] minus (119889119904
119910119889119905) [119903
4119887 + 1199036119886 + 120596119910 [
119886]2+ 120596119910 [
119887]2]
[119886]2+ [119887]2
= minus
119889119904119909
119889119905
120596119909minus
119889119904119910
119889119905
120596119910minus
119889119904119909
119889119905
sdot
1199033119887 + 1199035119886
[119886]2+ [119887]2minus
119889119904119910
119889119905
sdot
1199034119887 + 1199036119886
[119886]2+ [119887]2
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
v00(120596119909120596119910119905)
(A7)
Disclosure
The authorrsquos names are alphabetically listed
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgment
The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232
References
[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010
[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010
[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014
[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010
20 Computational Intelligence and Neuroscience
[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013
[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014
[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956
[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010
[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985
[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965
[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981
[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015
[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014
[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999
[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015
[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001
[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996
[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980
[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013
[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013
[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014
[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014
[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach
to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012
[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990
[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987
[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998
Submit your manuscripts athttpwwwhindawicom
Computer Games Technology
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Distributed Sensor Networks
International Journal of
Advances in
FuzzySystems
Hindawi Publishing Corporationhttpwwwhindawicom
Volume 2014
International Journal of
ReconfigurableComputing
Hindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Applied Computational Intelligence and Soft Computing
thinspAdvancesthinspinthinsp
Artificial Intelligence
HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014
Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Journal of
Computer Networks and Communications
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation
httpwwwhindawicom Volume 2014
Advances in
Multimedia
International Journal of
Biomedical Imaging
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
ArtificialNeural Systems
Advances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience
Industrial EngineeringJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Human-ComputerInteraction
Advances in
Computer EngineeringAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience 19
Now
119889119886
119889119905
(120596119909 120596119910 119905)
= int
R2
120597119906
120597119905
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
(A4)
For a purely translational signal with instantaneous spatialshift given by (119904
119909(119905) 119904119910(119905)) within the support of the window
we have
119889119886
119889119905
(120596119909 120596119910 119905)
= minusint
R2
119889119904119909
119889119905
(119905)
120597119906
120597119909
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus int
R2
119889119904119910
119889119905
(119905)
120597119906
120597119910
(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119909
119889119904119909
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
minus 120596119910
119889119904119910
119889119905
int
R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596
119909119909 + 120596119910119910) 119889119909 119889119910
=
119889119904119909
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199033=1199033(120596119909120596119910119905)
+ 120596119909
119889119904119909
119889119905
119887
+
119889119904119910
119889119905
int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199034=1199034(120596119909120596119910119905)
+ 120596119910
119889119904119910
119889119905
119887
(A5)
Similarly
119889119887
119889119905
(120596119909 120596119910 119905) = minus
119889119904119909
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119909
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199035=1199035(120596119909120596119910119905)
minus 120596119909
119889119904119909
119889119905
119886 minus
119889119904119910
119889119905
sdot int
R2119906 (119909 119910 119905)
120597119908
120597119910
(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
1199036=1199036(120596119909120596119910119905)
minus 120596119910
119889119904119910
119889119905
119886
(A6)
Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to
11988912060100
119889119905
(120596119909 120596119910 119905) =
minus (119889119904119909119889119905) [119903
3119887 + 1199035119886 + 120596119909 [
119886]2+ 120596119909 [
119887]2] minus (119889119904
119910119889119905) [119903
4119887 + 1199036119886 + 120596119910 [
119886]2+ 120596119910 [
119887]2]
[119886]2+ [119887]2
= minus
119889119904119909
119889119905
120596119909minus
119889119904119910
119889119905
120596119910minus
119889119904119909
119889119905
sdot
1199033119887 + 1199035119886
[119886]2+ [119887]2minus
119889119904119910
119889119905
sdot
1199034119887 + 1199036119886
[119886]2+ [119887]2
⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟
v00(120596119909120596119910119905)
(A7)
Disclosure
The authorrsquos names are alphabetically listed
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgment
The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232
References
[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010
[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010
[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014
[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010
20 Computational Intelligence and Neuroscience
[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013
[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014
[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956
[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010
[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985
[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965
[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981
[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015
[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014
[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999
[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015
[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001
[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996
[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980
[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013
[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013
[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014
[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014
[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach
to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012
[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990
[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987
[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998
Submit your manuscripts athttpwwwhindawicom
Computer Games Technology
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Distributed Sensor Networks
International Journal of
Advances in
FuzzySystems
Hindawi Publishing Corporationhttpwwwhindawicom
Volume 2014
International Journal of
ReconfigurableComputing
Hindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Applied Computational Intelligence and Soft Computing
thinspAdvancesthinspinthinsp
Artificial Intelligence
HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014
Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Journal of
Computer Networks and Communications
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation
httpwwwhindawicom Volume 2014
Advances in
Multimedia
International Journal of
Biomedical Imaging
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
ArtificialNeural Systems
Advances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience
Industrial EngineeringJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Human-ComputerInteraction
Advances in
Computer EngineeringAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
20 Computational Intelligence and Neuroscience
[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013
[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014
[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956
[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010
[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985
[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965
[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981
[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015
[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014
[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999
[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015
[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001
[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996
[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980
[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013
[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013
[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014
[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014
[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach
to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012
[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990
[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987
[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998
Submit your manuscripts athttpwwwhindawicom
Computer Games Technology
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Distributed Sensor Networks
International Journal of
Advances in
FuzzySystems
Hindawi Publishing Corporationhttpwwwhindawicom
Volume 2014
International Journal of
ReconfigurableComputing
Hindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Applied Computational Intelligence and Soft Computing
thinspAdvancesthinspinthinsp
Artificial Intelligence
HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014
Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Journal of
Computer Networks and Communications
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation
httpwwwhindawicom Volume 2014
Advances in
Multimedia
International Journal of
Biomedical Imaging
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
ArtificialNeural Systems
Advances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience
Industrial EngineeringJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Human-ComputerInteraction
Advances in
Computer EngineeringAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Submit your manuscripts athttpwwwhindawicom
Computer Games Technology
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Distributed Sensor Networks
International Journal of
Advances in
FuzzySystems
Hindawi Publishing Corporationhttpwwwhindawicom
Volume 2014
International Journal of
ReconfigurableComputing
Hindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Applied Computational Intelligence and Soft Computing
thinspAdvancesthinspinthinsp
Artificial Intelligence
HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014
Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Journal of
Computer Networks and Communications
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation
httpwwwhindawicom Volume 2014
Advances in
Multimedia
International Journal of
Biomedical Imaging
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
ArtificialNeural Systems
Advances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Computational Intelligence and Neuroscience
Industrial EngineeringJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Human-ComputerInteraction
Advances in
Computer EngineeringAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014