5lse0 mod 05 gaborvca.ele.tue.nl/sites/default/files/5lse0 mod 05 gabor wavelets_0.pdf · pdw-ap /...

1

1

Fac EE / SPS-VCAPdW-AP / 2019 5LSE0 / Module 05

Gabor wavelets & Hrchy. systems

Techniques for Video Compression and Analysis (5LSE0), Module 05

Gabor wavelets and hierarchical feedforward systems

Arash Pourtaherian([email protected])

2



Outline Introduction Space/Time & Frequency Gabor Functions

– 1D Gabor functions– 2D Gabor functions– 2D Gabor filter banks– Gabor uncertainty principle and limitations

Log-Gabor Filters

3



Introduction – (1) Among various wavelet bases, Gabor functions provide

the optimal resolution in both the time (spatial) and frequency domains

Gabor wavelets shown to be the optimal basis to extract local features for several applications in computer vision, i.e.:– Compression– Edge and line detection– Texture classification– Motion analysis and object recognition– etc.

4



Introduction – (2) Biological motivation

– The simple cells of the visual cortex of mammalian brains are best modeled by 2D Gabor wavelets

Hyvarinen, Hurri, Hoyer, 2009

Receptive fields of simple cells estimated by reverse correlation based on single-cell recordings in a macaque monkey

2

5



Introduction – (3) Mathematical and empirical motivation

– Gabor wavelet transform has both the multi-resolution and multi-orientation properties and are optimal for measuring local spatial frequencies

– Furthermore, Gabor wavelets are found to yield distortion tolerance space for pattern recognition tasks

6



Space/time & frequency – (1) The basic problem of Fourier transform for

signal/image processing– Recap:

• FT of any function that is bounded in time (space), has frequencies over an infinite range.

• Conversely, the inverse FT of a perfect band-pass filter has temporal (spatial) components over an infinite range.

– We may detect that a transient has occurred; but we cannot detect when!

7



Space/time & frequency – (2)– ‘rect’ function and its FT that

extends to infinity

– Perspective plot and image representing a 2D ideal HPF and its spatial representation

Inverse FT

8



Space/time & frequency – (3) There is an uncertainty relation between a

signals specificity in time and frequency

Dennis Gabor (1950) defined a family of signals that optimised this trade-off by choosing Gaussianwindowing functions

Time

Frequency

𝐹

𝑡 Time

Frequency

𝐹

𝑡

Good localization in time/space

Good localization in frequency

OR

3

9



Beyond wavelets Wavelets cannot optimally approximate

curvilinear singularities– Many coefficients are needed to

account for edges along curves *lets with directional sensitivity

– Ridgelets (Candès and Donoho, 1998)– Curvelets (Candès and Donoho, 2002)– Contourlets (Do and Vetterli, 2002)– Shearlets (Kutyniok and Labate, 2006)– etc.

10



Gabor functions – (1) Gabor showed that most

efficient (noiseless) channel divides time-frequency plane into cells of size: ∆𝑡∆𝑓 1

Introducing the Gabor expansion of signal 𝑠 as𝑠 𝑡 𝑎 , 𝑔 ,

, ∈𝕫

𝑔 , 𝑡 𝑒 𝑒 , 𝑡 𝑓 1

Time

Frequency

∆f

∆𝑡

11



Gabor functions – (2) Maximizing simultaneous resolution in time/space

and frequency

𝑡 𝑓 1

12



Gabor functions – (3) Functions 𝑔 , are called the Gabor elementary

functions and can be considered as the optimal time-frequency synthesizing elements

A Gabor elementary function is the modulation product of a harmonic oscillation with a Gaussian𝑔 𝑡 𝑒 . 𝑒

Gaussian envelope Harmonics

Gabor filter functions in time and frequency domains ut

4

13



2D Gabor functions – (1) 2D Gabor elementary functions

ℎ 𝑥, 𝑦1

2𝜋𝜎𝑒 . 𝑒

In the Fourier domain, the Gabor is a Gaussian centered about the central frequency (U,V)

The orientation of the Gabor in the spatial domain is 𝜃 tan

Symmetric Gaussian envelope Modulating Sine and Cosine

𝑢

𝑣

𝐻 𝑢, 𝑣 𝑒

14



2D Gabor functions – (2) Spatial frequency bandwidth

– Bandwidth at half power point

– Bandwidth depends reciprocally on symmetric Gaussian envelope’s sigma

Frequency 𝑈 𝑉

Am

plitu

de (

dB) 3dB

𝑏𝜎 1.09 𝑈 𝑉 ⇒ 𝑏 0.5

𝜎 0.56 𝑈 𝑉 ⇒ 𝑏 1

𝜎 0.31 𝑈 𝑉 ⇒ 𝑏 2

Spectral (Fourier)

Wide bandwidth Narrow bandwidth

Spatial

15



2D Gabor functions – (3) Spatial filter profile

Odd symmetric Sine Gabor waveletEven symmetric Cosine Gabor wavelet

𝜃 tan𝑉𝑈

45°

16



2D Gabor functions – (4) Gabor filter with asymmetric Gaussian

– An elliptical spatial Gaussian envelope can control the orientation bandwidth. A better formulation:

𝑔 𝑥, 𝑦𝑓

𝜋𝛾𝜂 𝑒 . 𝑒

𝑥 𝑥 cos 𝜃 𝑦 sin 𝜃𝑦 𝑥 sin 𝜃 𝑦 cos 𝜃

𝑓 central frequency𝜃 angle𝛾 sigma in direction of wave propagation𝜂 sigma perpendicular to direction of propagation

𝛾𝜂 1

𝛾𝜂 0.5

Symmetric Asymmetric

5

17



2D Gabor functions – (5) Orientation bandwidth

Ψ 𝑢, 𝑣 𝑒𝑢 𝑢 cos 𝜃 𝑣 sin 𝜃𝑣 𝑢 sin 𝜃 𝑣 cos 𝜃

Spectral domainSpatial domain

Daugman, 1985

𝜃

1𝜂

1𝛾

18



2D Gabor filter bank – (1)Using a filter bank with 𝑘 oriented filters, orientation bandwidth is chosen such that the half power profiles of the filters touch each other in the spectral domain

𝜂2𝑘𝜋 ln

12

𝑢

𝑣

𝑓

Orientation bandwidth

Spatial frequency bandwidth

19



2D Gabor filter bank – (2) Multi resolution analysis

– Logarithmic spacing of Gabor functions in the frequency domain, also plotted in time-frequency plane

Kämäräinen, 2003

20



2D Gabor filter bank – (3)Multi resolution analysis with filters at several scales

Half magnitude isolines of a Gabor filter bank with 2 scales and 8 orientations

van der Sommen, 2017

6

21



Uncertainty in 2D The Gabor filter in b is very localized in orientation In order to make the basic Gabor function in a more

localized in orientation, it is necessary to make it longer, and thus to reduce its spatial localization

22



Bandwidth limitations of Gabor filters– Non-orthogonal functions and therefore not reversible– If the standard deviation of Gabor Gaussians in the

frequency domain becomes more than about one third of the center frequency, the tails of the two Gaussians will start to overlap excessively at the origin, resulting in a nonzero DC component.

– Not possible to construct Gabor functions of arbitrarily wide bandwidth

23



Log-Gabor filtersDavid Field (1987) introduced that natural images can be better coded by filters that have Gaussian transfer functions when viewed on the logarithmic frequency scale

Ψ 𝜔 𝑒 / /

spatial domain

24



Log-Gabor filters – (2)– Log-Gabor filters have zero response at DC frequency– Log-Gabor filters, having extended tails, can encode

natural images more efficiently, compared to Gabor filters that tend to over-represents lower frequencies

7

25



Log-Gabor filters – (3)Construction of two-dimensional Log-Gabor filter

The two dimensional filter in the spectral domain consists of (a) a radial log-Gabor component and (b) an angular component, which form the final component (c).

26



Hierarchical Feedforward Systems –biologically inspired analysis

27



Outline 2D Gabor Representation in Vision HMAX Systems Hierarchical Feedforward Systems Spatiotemporal Gabor Filters Physiologically Plausible Filters

28



2D Gabor representation in vision Hubel and Wiesel (1960s)

– Certain neurons are excited only when a specific orientation of edges moves in the field of vision

– Hierarchical organization of neurons in the early visual cortex, starting to detect simple patterns, e.g. edges, followed by layers responding to more complex patterns

8

29



2D Gabor representation in vision – (2) Daugman (1985)

– Summarizes considerable physiological evidence that 2D Gabor functions are fundamental to visual processing in several mammalian species

– Visual channels have a frequency bandwidth of 1–2 octaves and total angular bandwidth of 30°

– The regions of sensitivity in the spectral domain are elliptical corresponding to:𝛾/𝜂 2

Spectral domainSpatial domain

30



2D Gabor representation in vision – (3)

Daugman, 1985

31



HMAX systemsHierarchical Model of object recognition in corteXSimple layers (S) of template matching, alternate with complex units (C) of max-pooling, thereby gradually increasing the complexity of the modeled structures

Low-level features Mid-level features

32



HMAX systems – (2)– S1: Gabor filters are used at multiple orientations to

extract low-level features, i.e. gratings, bars and edges– C1: Max-pooling locally over S1 responses to become

invariant to distortions and small shifts in the data

Size and shift tolerances are increased at C levels

Local max over S1 within band

9

33



Hierarchical feedforward systems – (1)– Cortical Network Simulator (CNS) Mutch et al., 2010

34



Hierarchical feedforward systems – (2)– Action recognition in streaming video Jhuang et al., 2007

35



Hierarchical feedforward systems – (3)– Convolutional neural networks

Szegedy et al., 2014GoogLeNet

LeNet-5LeCun et al., 1980

36



Hierarchical feedforward systems – (4)– Convolutional neural networks (cont’d)

10

37



Spatiotemporal Gabor filters – (1) Velocity equals distance over time or pixels per frame

– Motion is an oriented line or slab in space-time domain

Adelso and Bergen, 1985

38



Spatiotemporal Gabor filters – (2)– Spatial filters can be extended with a temporal component

for sensitivity to certain spatial and temporal frequencies– Filters oriented in space-time domain can detect a moving

stimulus. The orientation of the filter relates to its preferred speed and direction of motion

sampled motioncontinuous motion

39



Spatiotemporal Gabor filters – (3)– Electrophysiological studies indicate that the space-time

receptive-field structure of cortical simple cells is similar to that of 3-D Gabor filter

– Each neuron has a “spatiotemporal receptive field”X

TX

Y

De Angelis, Ohzawa, Freeman, 1993

Direction selective simple cell in the cat striate cortex (area 17)

40



Spatiotemporal Gabor filters – (4)An example of a space-time Gabor filter

An alternative formalization based on biological perspective– Primary visual cortex cells have a sharp tuning to motion with a

certain speed and direction (Petkov and Subramanian, 2007)

𝑔 𝑥, 𝑦, 𝑡1

2𝜋 / 𝜎 𝜎 𝜎𝑒 . 𝑒

𝑔 𝑥, 𝑦, 𝑡, 𝑣𝛾

2𝜋𝜎 𝑒 . cos2𝜋𝜆 𝑥 𝑣𝑡

11

41



Motion energy– Problem: Gabor filters are phase sensitive and sign of the

response depends on the stimulus contrast– Solution: Motion energy filters are constructed with a

quadrature pair: real and imaginary pair

42



Separable spatiotemporal filtersIf a unit gathers inputs from a set of spatially distributed positions, weights them by a spatial impulse response, and then sends the output through a temporal filter, the resulting spatiotemporal impulse response will be separableA spatiotemporal filter can be created as the product between a spatial filter and a temporal filter

Spatial impulse response H 𝑥Temporal impulse response H 𝑡Spatiotemporal impulse response H 𝑥, 𝑡 = H 𝑥 H 𝑡

43



Motion opponency – (1)Waterfall illusion: If you adapt to rightward motion, and then look at static image, you see leftward motion – The illusion demonstrates that perceived motion is

different from physical motion– We cannot see both left and right

motion at the same time. If you superimpose a leftward moving sinewave pattern on a rightward moving sinewave pattern, you don't see motion, you see flicker.

44



Motion opponency – (2)Psychophysical results suggest that neurons in the brain use a motion-opponent processing framework (e.g. left vs. right)– A simple opponent-motion channel

can be constructed by taking the arithmetic difference between the leftward and the rightward responses.

– This channel gives a positive output when there is rightward motion, a negative output for leftward motion, and no output for stationary patterns or for counter-phase flicker.

12

45



Other *lets and applications

RidgeletsCurvelets

ContourletsShearlets

46



Continuous ridgelet transform For each scale 𝑎 0, each position 𝑏 ∈ ℝ and each

orientation 𝜃 ∈ 0,2𝜋 , we define the bivariate ridgelet𝜓 , , : ℝ → ℝ by:

𝜓 , , 𝑥, 𝑦 𝑎 / . 𝜓 𝑥 cos 𝜃 𝑦 sin 𝜃 𝑏 /𝑎

A ridgelet example Rotation Dilation Translation

47



Discrete ridgelet transform The ridgelet coefficients are obtained by analysis of the

Radon transform of data on a Cartesian grid

Ridgelet spatial domain Ridgelet frequency domain

48



Curvelet transform – (1) Decomposition of the image into subbands Spatial partitioning (“windowing”) of each subband Applying the discrete ridgelet transform (DRT)

13

49



Curvelet transform – (2) Similarly to STFT, space localization is achieved by dividing

the signal into pieces by means of a “window” function

Curvelet spatial domain Curvelet frequency domain

50



Countourlet transform – (1) Multiscale decomposition

– Handled by a Laplacian pyramid Directional decompositions

– Handled by a directional filter bank.

51



Countourlet transform – (2)

Each image is decomposed into two pyramidal levels, which are then decomposed into four and eight directional subbands

52



Shearlet transform

Spatial domainShearlet frequency tiling

Frequency domain

14

53



Applications: Ridgelet transform Denoising

Original image containing a vertical band embedded

in white noise with relatively large amplitude

Denoised image using the wavelet transform

Denoised image using the Discrete Ridgelet

Transform (DRT)

Fadili and Starck, 2007

54



Applications: Curvelet transform Curve recoveryThe Picasso picture War and Peace (top left), the same image contaminated with a Gaussian white noise (top right). The restored images using the wavelet transform (bottom left) and the Discrete CurveletTransform (bottom right)


55



Curvelet transform: applications – (3) Contrast enhancement


Original Saturn image Result of curvelet-based contrast enhancement

56



Applications: Gabor wavelets – (1) Oncology

van der Sommen, 2017

15

57



Applications: Gabor wavelets – (2) Instrument localization in 3D ultrasound

58



Applications: Gabor wavelets – (3) Instrument localization in 3D ultrasound (cont’d)

59



Applications: log-Gabor wavelets Retinal image analysis(a) Input image, (b) Green channel, (c) Construction of orientation scores,(d) Candidate selection, (e) Classification to bifurcations (orange) and crossings (blue)

Sureshjani, 2017

60



Conclusions Space/Time & Frequency Gabor Functions Log-Gabor Filters Biologically inspired analysis Hierarchical Feedforward Systems Spatiotemporal Gabor Filters Physiologically Plausible Filters Other *lets and applications

5lse0 mod 05 gaborvca.ele.tue.nl/sites/default/files/5lse0 mod 05 gabor wavelets_0.pdf · pdw-ap /...

Documents