3 md-sampling s12 v5 2p

7/24/2019 3 MD-Sampling S12 v5 2p

1/18

13 -

D. Malah Video Signal Processing - Spring 2012

Multidimensional Sampling.3

Basics of Lattice Theory

Sampling Over Lattices

Sampling of Video Signals

Filtering Operations in Cameras and Display Devices

References

Wang et. al., Ch. 3.

Bovik, Ch. 2. (by E. Dubois)

E. Dubois, The Sampling and Reconstruction of Time-Varying Imagery with

Application in Video Systems, Proc. IEEE, Vol 73, No. 4, April 1985, pp. 502-522.

Tekalp, Ch.3.

23 -


Basics of Lattice Theory

As video sampling is done at points on a regular grid (not necessarily rectangular)

in the 3-D space, Lattice Theory can provide the mathematical foundation for this

operation.

Lattice - definition:

A lattice, ,in the real K-D space, , is the set of all possible vectors thatcan be represented as integer-weighted combinations of a set of K

linearly independent basis vectors,

K

1 2v , , , ,Kk k K

That is:

1

x x v ,K

Kk k k

k

Lattice n n

1 2V v ,v , ..., vKGenerating matrix:

1 2, K

Kn n n x Vn n [ , , ..., ]

7/24/2019 3 MD-Sampling S12 v5 2p

2/18

33 -


Examples of Lattices

1

1 0

0 1V =

Orthogonal Lattice1

3

22 1

2

0

1V =

Hexagonal Lattice

2

[Wang]

Unit Cells

1x

2x

43 -


Lattice Properties - 1

The basis vectors uniquely define a Lattice.

The basis vectors associated with a Lattice are non-unique.

Examples

3

2

3 12

3

0V =

3 2;

Any point on the grid of a Lattice is determined by a set of indices:

1 2n [ , , , ] K

Kn n n

Theorem: Unit Cell

For a given lattice , there exists a unit cell so that its translation to

all lattice point form a covering of the entire space :

K

( ifx

x) = x ( y x yK

; ,

where, is the translation of by x. ( |x p x p This is a tiling of by unit cells.

K

4

1 1

0 1

V =

4 1

7/24/2019 3 MD-Sampling S12 v5 2p

3/18

7/24/2019 3 MD-Sampling S12 v5 2p

4/18

73 -



Volume of a unit Cell

All unit cells of a given lattice have the same volume: det| V |

Sampling Density

Number of unit cells in a unit volume of :K 1det

() =| V |

d

Reciprocal Lattice

Hence: andT T T T I V I V U UV U V U

Let , and *x V m y Un then,

x,y x y m V Un m nT T T T

For a given lattice with a generating matrix , its reciprocal lattice

Is defined as a lattice with a generating matrix U, given by:

V *

1U (V )T .

Thus, the basis functions of V and U are biorthonormal:

1

;0,

,

v u ,,

T

k k

k

kk

Useful property

83 -


1

1 0

0 1V =

3

22 1

2

0

1V =

1

1 0

0 1U =

2 1

3 3

20 1

U =

1U (V )

T V

11() =d

11*( ) =d


Reciprocal Lattice Examples

2

2

3( ) =d 2

3

2

*( ) =d

1

2

1

*

2

*

1det*( ) = | V |

()d

d

1det

det| U |

| V |

2*( )

2( )

[Wang]

1

x

1x

2x 2f

2f

1

f

1f

7/24/2019 3 MD-Sampling S12 v5 2p

5/18

93 -


D Periodic Signal-Lattice of a KPeriodicity

1-D periodic signal:

; - non-singularx x Vn , n VK

,x x nT n

K-D periodic signal:

Hence, considering as the generating matrix of a lattice,

the particular lattice is called: Periodicity Lattice.

The Voronoi cell of the lattice is then called the Fundamental Period.

V

103 -


Sampling Over Lattices

The lattice structure provides a tool for uniform sampling of

multidimensional signals, but not necessarily on a rectangular (hypercube)

grid.

Sampling on a lattice

n Vn , n Ks c where,

- Continuous space signal,(x) x Kc

- Sampled space signal,(n) n Ks

Alternatively, Impulse Sampling:

n(x) n x Vn , xKK

s s

Sampled multi-

dimensional Signal

7/24/2019 3 MD-Sampling S12 v5 2p

6/18

113 -


Sampled Space Fourier Transform - SSFT

exp 2n

f n f VnK

Ts s j

SSFT:

Recall,

DSFT: exp 2n

(f ) (n) ( f n)K

T

sd j

CSFT: ( ) ( ) exp( 2 )f x f x xK

T

cc j d

If V = I, SSFT reduces to the DSFTRemark:

Discrete Space

F. T.

Continuous Space

F. T.

123 -


Since and , we get:

Periodicity of the SSFT

The SSFT is periodic: 1

;f Um f U VTs s

Thus, U, the periodicity lattice, is the generating matrix of the

reciprocal lattice and the fundamental period is the

Voronoi cell

*

*( )

Proof

f Vn : (f Um) Vn f Vn m U VnT T T T T

2n

f n exp f VnK

Ts s j

exp 2 1 ;( )j

exp 2 exp 2( (f Um) Vn) ( f Vn)T Tj j

TU V I

m nT

.f Vn m nT T

[Wang]

2

1f

7/24/2019 3 MD-Sampling S12 v5 2p

7/18

133 -


Inverse SSFT

1

exp 2*( )

n f f Vn f , n

T Ks s j d

d

*1

| det( ) |d Vol

V

Proof by:

exp 20

T d

j dotherwise

*( )

, n mf V(n m) f

,

where,

Linear convolution of signals sampled on the same lattice:

SSFT

K

s c s c

s s s s

s s s s

h h

h h

h H

m

n Vn ; n Vn

n * n n m m

n * n f f

Then,

Let:

143 -


Generalized Nyquist Theorem

If a continuous signal is sampled over a lattice , with a

Generating matrix V, then the SSFT of the sampled signal

, ,(x) Kc

x

n Vn , n Ks c

Is given in terms of the CSFT of the original continuous signal by:

SSFTm

f n f UmK

s s cd

And, it is possible to reconstruct the original signal from ,

Iff the support region of the CSFT of the original signal is limited within

the Voronoi cell of the reciprocal lattice, i.e.,

(x)c ns

0c for *f f The reconstruction can then be done by filtering the sampled signal

(impulse sampling) by a filter whose CSFT is:

1

0

*, f

,

r

fdH

otherwise

7/24/2019 3 MD-Sampling S12 v5 2p

8/18

153 -


[Wang]

fs

1f 1f

1f

1f

11

f

2f

2

2

2f

2

2f

1 3/r

1 2/

1 2/

1 2/

1 2/

1 2/

1 2/

1 2/

1 2/

163 -


Proof of the Generalized Nyquist Theorem

Since we can apply the Inv-CSFT to determine : n Vn ,s c ns

2(n) (V n) (f )exp( f Vn) f K

T

s c c j d

exp 2*m ( )

f Um ( (f Um) Vn) f K

Tc j d

Using: exp 2 exp 2 (since )T T T T T j j ( (f Um) Vn) ( f Vn) m U Vn m n

we get:

exp 2

exp 2

K

K

Ts c

Tc

j d

d

*

*

m ( )

m( )

(n) f Um ( f Vn) f

f Um ( f Vn) f

Comparing with the expression for the Inv-SSFT:

1

exp 2*( )

n f f Vn f , n

T Ks s j d

d

we find,

m

f f UmK

s cd

7/24/2019 3 MD-Sampling S12 v5 2p

9/18

173 -


Reconstruction by Interpolation

y

x x y y yK

r r sh d

n y

x n x y y Vn yK K

r s rh d

n

(y) n y Vn , yK

Ks s

Using impulse sampling:

Hence,

and we get the Interpolation Formula:

n

x n x VnK

r s rh

Reconstructed

Signal

183 -


Summary of Relations between Fourier Transforms

( ) ( ) exp( 2 )f x f x xK

T

cc j d

CSFT:

DSFT: 2n

(f ) (n)exp( f n)K

T

sd j

exp 2n

f n f VnK

Ts s j

SSFT:

Sampling: n Vn , n K

s c

1

Unit Hypercube

K

K

Ts d

Kd s

T Ts d d s

s c

Kd c

I

d

d I

*

*

m

m

f V f , f

f Uf , f

f Uf ; f V f U V

f f Um , f

f Uf Um , f

1 1f Uf V f ; f V f U f T Tc d d d c c

If is band-limited to within then:( )xc * f Uf , f Kd cd I Note

7/24/2019 3 MD-Sampling S12 v5 2p

10/18

193 -


Sampling Efficiency

1 (no aliasing)

*() =

( )

Vol Unit Sphere

Vol

Sampling Efficiency:

Assume that the support of signal spectrum is a unit sphere.

To avoid aliasing, should enclose the sphere.*( )

Definition

Since *( ) ()Vol d () =()

Vol Unit Sphere

d

The closer is to 1 (from below), the higher the effinciency()

Examples (2D)

Vol Unit Sphere

2D:

3D:43

[Wang]

4 4d ( ) ; ( ) / 4 4d ( ) ; ( ) / 2 3 2 3d ( ) ; ( ) /

203 -


Sampling of Video Signals

From the properties of the HVS

To avoid flicker: 60 to 70 frames/sec (orfps) (high illumination)

At a viewing distance of 3 times screen height:

3180 180

( )s s sf df f f cpdh

Thus, for 25 cpd (low visibility of lines), the spatial resolution required is:

180 18025 480

3 3( / )

sf f lines cycles picture height

and, for an image aspect ratio (IAR) of 4:3, we get 640 pixels/line

NTSC: 60 fields/sec (Interlaced), 240 active lines/ field

HDTV: IAR=16:9, 60 fps, 720 1080 lines/fr, 1280 1920 pixels/line

Computer Display: 72 fps (progressive), 1024 lines, 1280 pixels/line

7/24/2019 3 MD-Sampling S12 v5 2p

11/18

213 -


Video Raster Scan

Corresponds to Sampling of the 3D Video signal in the temporal and

vertical directions.

Thus, we consider sampling a continuous 2D video signal along the

temporal and vertical axes.

The Sampling Lattices for Progressive and Interlaced scans are:

Progressive:

Interlaced:

12

1 1 1

02 0= =

00V U

t

y

t

y

12

2 2 1 12

02

= =0V U

t

t y

t t

y

Where: - time between fields; - distance between consecutive linest y

223 -


11

12

1 1 1

02 0= =

00

T t

y

t

y

(V )V U

1

1

*

Progressive ScanNearest

aliasing

centers

[Wang]

(Fig. 3.6)

t

yf

7/24/2019 3 MD-Sampling S12 v5 2p

12/18

233 -


2

2

*

Interlaced ScanNearest

aliasing

centers

[Wang]

(Fig. 3.6)

1

2

1

22 2 1 1

2

02= =0

T

t

y y

t ty

(V )V U

t

243 -


Comparison of Interlaced and Progressive Scans

Same sampling density: 1 21 2

1 1 1

det det 2() = ( ) =

| V | | V |d d

t y

Same nearest aliases along the vertical frequency axis, at .1

y No motion: Same vertical resolution.

With motion, Interlaced has a closer alias, hence a lower resolution.

1

2 t In temporal direction, nearest alias for Progressive is closer, at ,

hence Interlaced has less flicker.

Different mixed (off-axis) aliases: for Progressive;

for Interlaced, resulting in Interline flicker for Interlaced.

1 1

2( , )

t y 1 1

2 2( , )

t y

For a signal with isotropic spectral support, the interlaced scan ismore efficient (assuming equal temporal and spatial frequency

scales).

7/24/2019 3 MD-Sampling S12 v5 2p

13/18

253 -


Sampling Video in 3D - 1

Assume frequency axes were calibrated., ,x y tf f f

Assume signal spectrum support in the normalized frequency

domain to be a Unit Sphere.

Progressive sampling intervals: , ,x y t

Aligned samples 3D Cube - Orthorhombic Lattice (ORT)

To avoid aliasing: 12

x y t 4388 6

( ) ; ( )d ORT ORT

Progressive

1

1

1

0 0

00

0 0

x

y

t

U=

[Wang]

(Fig. 3.7)

x

y

tfx

fy

ft

t1 / t

1 / x

1 /

263 -


Sampling Video in 3D 2

Interlaced sampling intervals: 2 2, , /x y t

Vertically aligned samples ALI Lattice

To avoid aliasing: 1 12 3;x y t 434 3

4 3 3 3( ) ; ( )d ALI ALI

Interlaced - 1

[Wang]

(Fig. 3.7)

1

12

1 2

0 0

00

0

U=

x

y

t t

x

y

t

fx

fy

ft

2/t

7/24/2019 3 MD-Sampling S12 v5 2p

14/18

273 -




Horizontally shifted samples by BCO Lattice

To avoid aliasing: 1 12 2 2

;x t y 4 23 2

( ) ; ( )d BCO BCO

Interlaced - 2

2/x

[Wang]

(Fig. 3.7)

x

y

t

fx

fy

ft

283 -




each field contains all the lines, but the samples in the same

field are interleaved in the even and odd lines

FCO Lattice To avoid aliasing: 1 1

2 2;x y t 4 2

3 2( ) ; ( )d FCO FCO

Interlaced - 3

Same as BCO, but better visually.

[Wang]

(Fig. 3.7)

xy

tfx

fy

ft

7/24/2019 3 MD-Sampling S12 v5 2p

15/18

293 -


[Wang]


Summary

303 -


1-On Spatial and Temporal Aliasing

Moving Sinusoidal Pattern (vertical lines)

1 /h

f cycle cm

Q. The pattern is moving horizontally at the speed of 3 cm/sec.

If we sample the pattern at:

What is the apparent motion and sinusoidal frequency?

3 3 sec, , ,

/ ; /s x s y s tf f samples cm f frames

Qualititative answer: On the camera image plane well get a stationary image.

1 0 0( , , ) ( , , )h v t

f f f

[Wang](Fig. 3.8)

7/24/2019 3 MD-Sampling S12 v5 2p

16/18

313 -


Analytic Answer

Applying the CSFT:

1 0 1 0

3 3

( , ) ( , ), ( , )

( ) ,

x y

t x x y y

f f

f f v f v

Thus, the 3D CSFT has a pair of impulses at: 1 0 3 1 0 3( , , ) ( , , ),( , , )x y tf f f

1-1

3

3-

xf

tftf

x

f 3, , ,s x s y s t

f f f

,: 2 6s t tAliasing f < f

1 0 0 1 0 0

:

( , , ) ( , , ), ( , , )x y t

Aliased components at

f f f

2-On Spatial and Temporal Aliasing

[Wang](Fig. 3.8)

tf

323 -


Filtering Operations in Cameras and Display Devices

Camera Aperture

, ,x y t Consider a cubic sampling lattice at intervals:

sampling frequencies, , ,

1 1 1, ,s x s x s tf f f

x y t

Ideal pre-filter would be a lowpass filter with cutoff frequenciesat half the sampling frequencies.

Practically we have the following image acquisition characteristics:

Temporal Aperture

- Integration over exposure time , corresponding to a temporal filter:e

,

1, [0, ]

( )

0,p t

t eh t e

otherwise

CSFT ,sin( )

( ) exp( ) tp t t tt

f eH f j f e

f

Thus, the cutoff freq. is . Since usually , we get aliasing.1

e e t

This is preferred on the blurring due to selection of a larger .e

7/24/2019 3 MD-Sampling S12 v5 2p

17/18

333 -


Spatial Aperture

Spatial sensor integration is modeled by:

2 2 2

, ,

1( , ) exp( ( ) / 2 )2

p x yh x y x y

2 2 2, , ,( ) exp( ( ) / 2 )

CSFT

p x y x y x yH f f f f

1

2

Is selected so that the normalized filters response is 0.5 at half

the sampling frequencies (horizontal & vertical). Hence for:

, ,

, , , ,2 ln 2

s x y

s x s y s x y

ff f f

Combined Aperture

, , ,( , , ) ( ) ( , ) CSFT

p p t p x yh x y t h t h x y , , , ,( , ) ( ) ( , )p x y t p t t p x y x yH f f f H f H f f

343 -


,1 ; 480 / .60

s ye t f lines pic height

( , )ph y t

( , )p y tH f f

tf

f

t

y

Combined Aperture

( 0)x

At f

[Wang]

7/24/2019 3 MD-Sampling S12 v5 2p

18/18

353 -


Comments

Human Vision Considerations

Human viewer tolerates more aliasing than resolution

loss (blurring).

The preservation of the signal in the pass-band is more important

to the viewer than attenuation outside the pass-band.

Digital cameras may capture at higher sampling rates and then

implement explicit filtering before converting to lower resolution.

Digital Filtering

In a CRT, a beam scans a phosphor screen. Width of beam

determines vertical filtering. A Wide beam causes blurring, hence a

thin beam is preferred (viewer may observe scan lines).

Temporal filtering is determined by phosphor response decay time.

HVS performs temporal interpolation

Note: Combination of camera and display apertures results in a maximum

vertical resolution that is only K=0.7 of the theoretical limit

(K - Kell factor).

Display Aperture

3 md-sampling s12 v5 2p

Documents