image processing - ch 05 - image restorationbarner/courses/eleg675/image processing...1 image...

1

Image Restoration

Image Processing with Biomedical Applications

ELEG-475/675Prof. Barner

Image ProcessingImage Restoration

Prof. Barner, ECE Department, University of Delaware 2

Image Restoration

Image enhancement is subjective Heuristic and ad hoc

Image restoration is more theoretically motivated

A priori knowledge of image degradation utilizedOptimality criteria used to formulate restoration



Preliminaries:Correlation and Power Spectrum

Note self-convolution

Correlation results if we do not reflect one term

Note maximum at Τ=0; also

Power Spectral Density (PSD)

Cross correlation and PSD

Assume ergotic signals (RVs) – interchange time/ensemble averages, e.g.,

( ) ( ) ( ) ( )f t f t f t f t dtτ∞

−∞∗ = −∫

( ) ( ) ( ) ( ) ( )fR f t f t f t f t dtτ τ∞

−∞= ∗ − = +∫

2

( ) ( )fR d f t dtτ τ∞ ∞

−∞ −∞

⎡ ⎤= ⎢ ⎥⎣ ⎦∫ ∫

{ } { } 2( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) | ( ) |f fP s R f t f t F s F s F s F s F sτ ∗= ℑ = ℑ ∗ − = − = =

( ) ( ) ( ) ( ) ( )fgR f t g t f t g t dtτ τ∞

−∞= ∗ − = +∫

( ) { ( )}fg fgP s R τ= ℑ

{ ( )} ( )x t x t dtε∞

−∞= ∫



Degradation and Restoration Model

Degradation is taken to be a linear spatially invariant operator

( , ) ( , ) ( , ) ( , )g x y h x y f x y x yη= ∗ +

( , ) ( , ) ( , ) ( , )G u v H u v F u v N u v= +

2



Noise Properties

Arises in acquisition, digitization, and transmission/storage processesCCD cameras are affected by:

Light levelsSensor temperatureBad sensors

Transmission noise can be due to interferenceWireless transmission interferenceLost networking packets

Statistics depend on sourceCommon assumption: White Spectrum

Correlation: R(Τ)=Aδ(Τ)Power spectrum: P(s)=A



Noise Probability Density Functions

Gaussian (normal) PDF

Typically models electronics and sensor noiseRayleigh PDF

Skewed distribution typically models range imaging noise

2 2( ) / 21( )2

zp z e μ σ

πσ− −=

2( ) /2 ( ) for

0 for ( )

z a bz a e z ab

z ap z

− −− ≥

<

⎧= ⎨⎩

2 (4 )/ 4, 4

ba b πμ π σ −= + =



Noise Probability Density Functions II

Gamma PDF

Exponential PDF

Both appropriate for laser imagingHeavy-tailed distributions

samples contain frequent outliers

1( ) for 0

( 1)!

0 for 0( )

b baza z z a e z

b

zp z

−−− ≥

−

<

⎧⎪= ⎨⎪⎩

22, b b

a aμ σ= =

{ for 00 for 0( )

azae zzp z

− ≥<=

22

1 1, a a

μ σ= =



Noise Probability Density Function III

Uniform PDF

Impulsive (salt and pepper) PDF

Shot or spike notice appropriate faulty sensor or electronics, transmission error/drop

1 if

0 otherwise( )

a z bb ap z

≤ ≤−

⎧= ⎨⎩

22 ( ),

2 12a b b aμ σ+ −

= =

for ( ) for

0 otherwise

a

b

P z ap z P z b

=⎧⎪= =⎨⎪⎩

3



PDF Plots

Gaussian distribution is most widely used

Central Limit TheoremDesirable propertiesIndependence and correlated

Other distributions appropriate for specific cases

Simplicity of uniform enables derivation of results



Test Pattern Corruption Example



Test Pattern Corruption Example II



Noise Parameter Determination

Generate statistics from uniform regionHistogram matchingQuantile-Quantile (Q-Q) plot

Determined defining statistics, e.g.2 2( ), ( ) ( )

i i

i i i iz S z S

z p z z p zμ σ μ∈ ∈

= = −∑ ∑

4



Q-Q Plot ExampleAre samples x1, x2,…, xN, and y1, y2,…, yN governed by the same distribution?

Order the samples: x(1), x(2),…, x(N), and y(1), y(2),…, y(N)

Plot (x(1), y(1)), (x(2), y(2)),…,(x(N), y(N))Samples governed by the same distribution will lie (approximately) along a line



Additive Noise Degradation Case

Gaussian case: mean (averaging) filter is ML optimal (previously derived)Laplacian (double exponential) case: median filter is ML optimal (previously derived)Both assume iid noise, constant signal

See book for Sample filtering resultsAdditional simple spatial filters

We consider arbitrary noise PSD



The Wiener Estimator

Assume no channel distortionDerived in one dimension for simplicityMean Square Error (MSE) optimality criteria:

h(t) ( )y t( )s t ( )x t

( )n t

2 2MSE { ( )} ( )

( ) ( ) ( )

e t e t dt

e t s t y t

ε∞

−∞= =

= −∫



Rearrangement of MSE

Calculus of variations approach:Obtain a functional expression of MSE in terms of h(t)Obtain an expression for the optimal impulse response, ho(t), in terms of known power spectraDevelop an expression for the MSE that results when ho(t) is used

Rewriting the MSE:2 2 2 2

2 21 2 3

MSE { ( )} {[ ( ) ( )] } { ( ) 2 ( ) ( ) ( )}MSE { ( )} 2 { ( ) ( )} { ( )}

e t s t y t s t s t y t y ts t s t y t y t T T T

ε ε ε

ε ε ε

= = − = − +

= − + = + +

5



MSE Components

Consider the first component:

Express the second component by writing the output as a convolution

2 21 { ( )} ( ) (0)sT s t s t dt Rε

∞

−∞= = =∫

{ }{ }

2

2

2

2 ( ) ( ) ( )

2 ( ) ( ) ( )

2 ( ) ( )xs

T s t h x t d

T h s t x t d

T h R d

ε τ τ τ

τ ε τ τ

τ τ τ

∞

−∞

∞

−∞

∞

−∞

= − −

= − −

= −

∫

∫∫



MSE Components II

Express the third component as the product of two convolutions:

utilizing change of variables: v=t-u

which is the autocorrelation evaluated at u-Τ

Final result:

Function of: filter impulse response, known autocorrelation and crosscorrelation

{ }{ }

3

3

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )

T h x t d h u x t u du

T h h u x t x t u d du

ε τ τ τ

τ ε τ τ

∞ ∞

−∞ −∞

∞ ∞

−∞ −∞

= − −

= − −

∫ ∫

∫ ∫

{ } { }( ) ( ) ( ) ( )x t x t u x v u x vε τ ε τ− − = + −

3 ( ) ( ) ( )xT h h u R u d duτ τ τ∞ ∞

−∞ −∞= −∫ ∫

MSE (0) 2 ( ) ( ) ( ) ( ) ( )s xs xR h R d h h u R u d duτ τ τ τ τ τ∞ ∞ ∞

−∞ −∞ −∞= − + −∫ ∫ ∫



MSE Minimization (I)

Define arbitrary impulse response:

inserting into MSE expression

First three terms: optimal MSE (MSEo)Autocorrelation Rx(u-Τ) is an even function

Fourth and fifth terms are equalCombine fourth, fifth, and sixth terms

( ) ( ) ( )oh t h t g t= +

MSE (0) 2 [ ( ) ( )] ( ) [ ( ) ( )][ ( ) ( )] ( )s o xs o o xR h g R d h g h u g u R u d duτ τ τ τ τ τ τ τ∞ ∞ ∞

−∞ −∞ −∞= − + + + + −∫ ∫ ∫

MSE (0) 2 ( ) ( ) ( ) ( ) ( )

( ) ( ) ( ) ( ) ( ) ( )

2 ( ) ( ) ( ) ( ) ( )

s o xs o o x

o x o x

xs x

R h R d h h u R u d du

h g u R u d du h u g R u d du

g R d g g u R u d du

τ τ τ τ τ τ

τ τ τ τ τ τ

τ τ τ τ τ τ

∞ ∞ ∞

−∞ −∞ −∞

∞ ∞ ∞ ∞

−∞ −∞ −∞ −∞

∞ ∞ ∞

−∞ −∞ −∞

= − + −

+ − + + −

− + −

∫ ∫ ∫∫ ∫ ∫ ∫∫ ∫ ∫



MSE Minimization (II)

Show T5 is nonnegativeExpanding autocorrelation

Define z(t)= g(t)*x(t)

Independent of ho

4 5

MSE MSE 2 ( ) ( ) ( ) ( ) ( ) ( ) ( )

MSE

o o x xs x

o

g u h R u d R u du g g u R u dud

T T

τ τ τ τ τ τ∞ ∞ ∞ ∞

−∞ −∞ −∞ −∞

⎡ ⎤= + − − + −⎢ ⎥⎣ ⎦= + +

∫ ∫ ∫ ∫

5

5

( ) ( ) ( )

( ) ( ) ( ) ( )

T g g u x t dtdud

T g x t u du g x t dtd

τ τ τ

τ τ τ τ

∞ ∞ ∞

−∞ −∞ −∞

∞ ∞ ∞

−∞ −∞ −∞

= −

= − −

∫ ∫ ∫∫ ∫ ∫

25 ( ) 0T z t dt

∞

−∞= ≥∫

6



MSE Minimization (III)

Expression to minimize

Necessary and sufficient condition:Make term in brackets 0 for all u

Note that for linear systems: [equation 48]Since the above is a convolution

In terms of PSD

5MSE MSE 2 ( ) ( ) ( ) ( )o o x xsg u h R u d R u du Tτ τ τ∞ ∞

−∞ −∞

⎡ ⎤= + − − +⎢ ⎥⎣ ⎦∫ ∫

( ) ( ) ( )xs o xR h u R u duτ τ∞

−∞= −∫

( ) ( ) ( ) ( )xs o x xyR h u R u Rτ τ= ∗ =

( ) ( ) ( ) ( )xs o x xyP s H s P s P s= =( )( )( )

xso

x

P sH sP s

=



Wiener Filter Design

To determine

Obtain an observation signal sample (x(t))Determine autocorrelation and PSD

Obtain a signal sampled without noise (s(t))Cross correlate x(t) and s(t); determine PSD

Implement filter in the frequency domain or take inverse transform to get impulse response

If statistics are unknown or can’t be measured, assume functional form

Example: white PSD

( )( )( )

xso

x

P sH sP s

=



Uncorrelated Signal and Noise Case

In this case: Crosscorrelation:

Autocorrelation:

Optimal filter:

{ } { } { }( ) ( ) ( ) ( )s t n t s t n tε ε ε=

{ } [ ]{ }( ) ( ) ( ) ( ) ( ) ( )xsR x t s t s t n t s tτ ε τ ε τ= + = + +

{ } { }( ) ( ) ( ) ( ) ( )xsR s t s t n t s tτ ε τ ε τ= + + +

{ } { }( ) ( ) ( ) ( ) ( ) ( ) ( )xs s sR R n t s t R n t dt s t dτ τ ε ε τ τ τ τ∞ ∞

−∞ −∞= + + = + +∫ ∫

( ) ( ) (0) (0)xs sR R N Sτ τ= +

( ) ( ) ( ) 2 (0) (0)x s nR R R N Sτ τ τ= + +

( ) (0) (0) ( )( )( ) ( ) 2 (0) (0) ( )

so

s n

P s N S sH sP s P s N S s

δδ

+=

+ +( )( )

( ) ( )s

os n

P sH sP s P s

=+



Analysis of Optimal MSE

Recall: For the uncorrelated case (see previous)

Further assuming zero means

Substituting for the optimal filter (even function)

MSE (0) ( ) ( )o s o xsR h R dτ τ τ∞

−∞= − ∫

( ) ( ) (0) (0)xs sR R N Sτ τ= +

{ }1MSE (0) ( ) ( )o s o sR h P s dτ τ∞ −

−∞= − ℑ∫

2MSE (0) ( ) ( ) j so s s oR P s h e d dsπ ττ τ

∞ ∞

−∞ −∞= − ∫ ∫

MSE ( ) ( ) ( )o s s oP s ds P s H s ds∞ ∞

−∞ −∞= − −∫ ∫

( )MSE ( ) ( )( ) ( )

so s s

s n

P sP s ds P s dsP s P s

∞ ∞

−∞ −∞= −

+∫ ∫( ) ( )MSE ( ) ( )

( ) ( )s n

o n os n

P s P s ds P s H s dsP s P s

∞ ∞

−∞ −∞= =

+∫ ∫

7



Wiener Filter Examples (I)

The Wiener filter transfer function Bandlimited signal



Wiener Filter Examples (II)

Separable signal and noise case

MSE is zero



Wiener Deconvolution

Combine deconvolution (system inversion) and noise filteringCombination of two linear operations

Spectrum of observed signal: Input to Wiener filter: For uncorrelated signal and noise (previous result)

Wiener (MSE optimal) deconvolution filter:

( ) ( ) ( ) ( )X s F s S s N s= +( )( ) ( ) ( ) ( )( )

N sY s S s S s K sF s

= + = +

2

22

( ) | ( ) |( )( ) ( ) ( )| ( ) |

( )

so

s k

P s S sH sP s P s N sS s

F s

= =+

+

( ) ( ) ( ) ( )1( )( ) ( ) ( ) ( ) | ( ) | ( ) ( )o s s

s k s n

H s P s F s P sG sF s F s P s P s F s P s P s

∗⎡ ⎤= = =⎢ ⎥+ +⎣ ⎦

( )F s1( )F s

( )oH s( )s t ( )w t ( )x t

( )n t

( )y t ( )z t

( )G s



Wiener Deconvolution Examples

Wiener deconvolutionexample: (a) signal and noise power spectra; (b) blurring function; (c) Wiener deconvolutiontransfer function

Example of two-dimensional Wiener deeconvolution: (a) signal power spectrum; (b) noise power spectrum; (c) blurring function; (d) transfer function

8



Wiener Generalization (I)

The two-dimensional Wiener filter:

Often used approximation:

2

2

1 | ( , ) |ˆ ( , ) ( , )( , ) | ( , ) | ( , ) / ( , )f

H u vF u v G u vH u v H u v S u v S u vη

⎡ ⎤= ⎢ ⎥

+⎢ ⎥⎣ ⎦

2

2

1 | ( , ) |ˆ ( , ) ( , )( , ) | ( , ) |

H u vF u v G u vH u v H u v K⎡ ⎤

= ⎢ ⎥+⎣ ⎦



Wiener Generalization (II)

Geometric Mean Filter generalization:

Special casesα=1: inverse filterα=0: parametric of Wiener filter (standard if β=1)α=1/2; β=1: referred to as the Spectrum equalization filter

The product of two quantities raised to the same powerDefinition of the geometric mean

Trade-off between inverse filtering and Wiener filtering controlled by α

1

22

( , ) ( , )ˆ ( , ) ( , )| ( , ) | ( , )

| ( , ) |( , )f

H u v H u vF u v G u vH u v S u v

H u vS u v

α

α

ηβ

−

∗ ∗

⎡ ⎤⎢ ⎥

⎡ ⎤ ⎢ ⎥= ⎢ ⎥ ⎢ ⎥⎡ ⎤⎣ ⎦ ⎢ ⎥+ ⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦⎣ ⎦



Degradation Function Estimation

Recall: In the no (low) noise case

Estimation by observation:

Estimation by experimentation:Input impulse of strength A

Estimation by modeling:Example: atmospheric interference

( , ) ( , ) ( , ) ( , )G u v H u v F u v N u v= +

( ,. )( , ) ˆ ( , )s

ss

G u vH u vF u v

=

( , )( , ) G u vH u vA

=

2 2 5 / 6( )( , ) k u vH u v e− +=



Estimation by Experiment Example

Experimentally determined point spread function (PSF)

9



Modeling Image Motion

Systems most often consider motionCamera and/or subject motion

If T is the exposure duration:

Blurred image: g(x,y)Planar motion trajectories: xo(t) and yo(t)

Fourier transform evaluation yields

where

[ ]0 00( , ) ( ), ( )

Tg x y f x x t y y t dt= − −∫

( , ) ( , ) ( , )G u v H u v F u v=

[ ]0 02 ( ) ( )

0( , )

T j ux t vy tH u v e dtπ− += ∫Image ProcessingImage Restoration


Motion Example

If xo(t)=at/T and yo(t)=0

Two dimensional linear (xo(t)=at/T and yo(t)=bt/T) motion

02 ( ) 2 /

0 0( , ) sin( )

T Tj ux t j uat T j uaTH u v e dt e dt ua eua

π π πππ

− − −= = =∫ ∫

[ ] ( )( , ) sin ( )( )

j ua vbTH u v ua vb eua vb

πππ

− += ++



Atmospheric Turbulence Degradation



Inverse Filtering Example

Direct application of

Amplifies noisePossible solution:

Limit cutoff frequency

( , )ˆ ( , )( , )

G u vF u vH u v

=

10



Inverse and Wiener Filtering Comparison

Wiener parameter K set experimentallyWiener result much sharper than the band limited inverse filterFull inverse filter results is useless



Motion Blur with Additive Noise Example



Mean in Geometric Mean Example



Multiple Applications of the Median

11



Mixed NoiseExample

Image corrupted byUniform noiseUniform and salt and pepper noise

Filtering methodsArithmetic meanGeometric meanMedianAlpha-trimmed mean



Mean, Arithmetic Mean, and Adaptive Approach



Median and Adaptive Median



Periodic Interference

Band reject filtersStraightforward extension from a high/low pass case

Ideal, Butterworth, Gaussian

12



Sinusoidal Corruption Example



Periodic Scan LineInterference

Note horizontal scan line “striping”

Vertical spectral comment

Notch pass filter and spatial interferenceImage with interference rejected



NASA Image, Spectrum, and Filtering Result

image processing - ch 05 - image restorationbarner/courses/eleg675/image processing...1 image...

Documents