digital image processing - Πανεπιστήμιο...
TRANSCRIPT
University of Ioannina - Department of Computer Science and Engineering
Image Restoration and Reconstruction
(Linear Restoration Methods)
Christophoros Nikou
Digital Image Processing
2
C. Nikou – Digital Image Processing (E12)
Contents
In this lecture we will look at linear image
restoration techniques
– Differentiation of matrices and vectors
– Linear space invariant degradation
– Restoration in absence of noise
• Inverse filter
• Pseudo-inverse filter
– Restoration in presence of noise
• Inverse filter
• Wiener filter
• Constrained least squares filter
3
C. Nikou – Digital Image Processing (E12)
Differentiation of Matrices and
Vectors
Notation:
A is a MxN matrix with elements aij.
x is a Nx1 vector with elements xi.
f(x) is a scalar function of vector x.
g(x) is a Mx1 vector valued function of vector x.
4
C. Nikou – Digital Image Processing (E12)
Differentiation of Matrices and
Vectors (cont...)
Scalar derivative of a matrix.
A is a MxN matrix with elements aij.
111
1
N
MNM
aa
t t
taa
t t
A
5
C. Nikou – Digital Image Processing (E12)
Differentiation of Matrices and
Vectors (cont...)
Vector derivative of a function (gradient).
x is a Nx1 vector with elements xi.
f(x) is a scalar function of vector x.
1
T
N
f f ff
x x
x
x
6
C. Nikou – Digital Image Processing (E12)
Differentiation of Matrices and
Vectors (cont...)
Vector derivative of a vector (Jacobian):
x is a Nx1 vector with elements xi.
g(x) is a Mx1 vector valued function of vector x.
1 1
1
1
N
M M
N
g g
x x
g g
x x
g
x
7
C. Nikou – Digital Image Processing (E12)
Differentiation of Matrices and
Vectors (cont...)
Some useful derivatives.
x is a Nx1 vector with elements xi.
b is a Nx1 vector with elements bi.
It is the derivative of the scalar valued function
bTx with respect to vector x.
T
b x b
x
8
C. Nikou – Digital Image Processing (E12)
Differentiation of Matrices and
Vectors (cont...)
Some useful derivatives.
x is a Nx1 vector with elements xi.
A is a NxN matrix with elements aij.
If A is symmetric:
T T
x Ax A A x
x
2T
x Ax Ax
x
9
C. Nikou – Digital Image Processing (E12)
Differentiation of Matrices and
Vectors (cont...)
Some useful derivatives.
x is a Nx1 vector with elements xi.
b is a Mx1 vector with elements bi.
A is a MxN matrix with elements aij.
It may be proved using the previous properties.
2
2 T
Ax b A Ax b
x
10
C. Nikou – Digital Image Processing (E12)
Linear, Position-Invariant
Degradation
We now consider a degraded image to be
modelled by:
where h(x, y) is the impulse response of the
degradation function ( i.e. point spread function
blurring the image).
The convolution implies that the degradation
mechanism is linear and position invariant (it
depends only on image values and not on
location).
( , ) ( , ) ( , ) ( , )g x y h x y f x y x y
11
C. Nikou – Digital Image Processing (E12)
Linear, Position-Invariant
Degradation (cont...)
In the Fourier domain:
where multiplication is element-wise.
In matrix-vector form:
where H is a doubly block circulant matrix and
f,g, and η are vectors (lexicographic ordering).
( , ) ( , ) ( , ) ( , )G k l H k l F k l N k l
g = Hf + η
12
C. Nikou – Digital Image Processing (E12)
Linear, Position-Invariant
Degradation (cont...)
If the degradation function is unknown the
problem of simultaneously recovering f(x,y)
and h(x,y) is called blind deconvolution.
g = Hf + η
( , ) ( , ) ( , ) ( , )g x y h x y f x y x y
( , ) ( , ) ( , ) ( , )G k l H k l F k l N k l
13
C. Nikou – Digital Image Processing (E12)
Estimating the point spread function
• In what follows, we consider that the degradation
function is known.
• If the psf is not known, some basic methods to
estimate it are:
– By observation
• Apply sharpening filters to a sub-image gs(m,n) where the
signal is strong (there is almost no noise) and obtain a visually
pleasant result fs(m,n). The psf may be approximated by
Hs(k,l)= Gs(k,l)/ Fs(k,l).
• The task needs trial and error and may be tedious.
• Used in special circumstances (e.g. Restoration of old
photographs)
14
C. Nikou – Digital Image Processing (E12)
Estimating the point spread function
(cont.)
• If the psf is not known, some basic methods to
estimate it are:
– By experimentation
• If the acquisition equipment or a similar one is available an
image similar to the degraded may be obtained by varying the
system settings.
• Then obtain the image of an impulse (small dot of light) using
the same settings.
15
C. Nikou – Digital Image Processing (E12)
Estimating the point spread function
(cont.)
• If the psf is not known, some basic methods to
estimate it are:
– By modeling: atmospheric turbulence:
5/62 2( , ) expH u v k u v
16
C. Nikou – Digital Image Processing (E12)
Estimating the point spread function
(cont.)
• By modeling: planar motion
– x0(t) and y0(t) are the time varying components of
motion at each pixel.
– The total exposure at any pixel is obtained by
integrating the instantaneous exposure over the time
the shutter is open.
– Assumption: the shutter opening and closing is
instantaneous.
– If T is the duration of the exposure, the recorded image
is expressed by:
0 00
( , ) [ ( ), ( )]T
g x y f x x t y y t dt
17
C. Nikou – Digital Image Processing (E12)
Estimating the point spread function
(cont.)
0 0
0 0
2 ( )
2 ( )
0 00
2 ( )
0 00
2 ( ) ( )
0
2 ( ) ( )
( , ) ( , )
[ ( ), ( )]
[ ( ), ( )]
( , )
( , )
j ux vy
Tj ux vy
Tj ux vy
T j ux t vy t
j ux t vy t
G u v g x y e dxdy
f x x t y y t dt e dxdy
f x x t y y t e dxdy dt
F u v e dt
F u v e
0 0
0
2 ( ) ( )
0( , )
T
T j ux t vy t
dt
H u v e dt
18
C. Nikou – Digital Image Processing (E12)
Estimating the point spread function
(cont.)
• Considering uniform linear motion:
0 0( ) , ( )t t
x t a y t bT T
• The psf becomes:
2 /
0( , )
sin
T j uat vbt T
j ua vb
H u v e dt
Tua vb e
ua vb
19
C. Nikou – Digital Image Processing (E12)
Estimating the point spread function
(cont.)
• Result of blurring with:
0 0( ) , ( ) , 0.1, 1t t
x t a y t b a b TT T
20
C. Nikou – Digital Image Processing (E12)
Linear Restoration
Using the imaging system
we want to estimate the true image from the
degraded observation with known
degradation H.
A linear method applies an operator (a
matrix) P to the observation g to estimate the
unobserved noise-free image f:
f̂ = Pg
g = Hf + η
21
C. Nikou – Digital Image Processing (E12)
Restoration in Absence of Noise
The Inverse Filter
When there is no noise:
an obvious solution would be to use the
inverse filter:
yielding
g = Hf
-1P = H
ˆ -1 -1f = Pg = H g = H Hf = f
22
C. Nikou – Digital Image Processing (E12)
Restoration in Absence of Noise
The Inverse Filter (cont...)
For a NxN image, H is a N2xN2 matrix!
To tackle the problem we transform it to the
Fourier domain.
H is doubly block circulant and therefore it
may be diagonalized by the 2D DFT matrix W:
ˆ -1f = H g
-1H = W ΛW
23
C. Nikou – Digital Image Processing (E12)
Restoration in Absence of Noise
The Inverse Filter (cont...)
where
Therefore:
-1H = W ΛW
{ (1,1),..., ( ,1), (1,2),... ( , )}diag H H N H H N NΛ
f̂ = Pg ˆ -1f = H g ˆ -1 -1f = (W ΛW) g
ˆ -1 -1f = W Λ Wg ˆ -1 -1
Wf = WW Λ Wg
ˆ -1F = Λ G
24
C. Nikou – Digital Image Processing (E12)
Restoration in Absence of Noise
The Inverse Filter (cont...)
This is the vectorized form of the DFT of the
image:
Take the inverse DFT and obtain f(m,n).
Problem: what happens if H(k,l) has zero
values?
Cannot perform inverse filtering!
ˆ -1F = Λ G( , )ˆ ( , )( , )
G k lF k l
H k l =
25
C. Nikou – Digital Image Processing (E12)
Restoration in Absence of Noise
The Pseudo-inverse Filter
A solution is to set:
which is a type of pseudo-inversion.
Notice that the signal cannot be restored at
locations where H(k,l)=0.
( , ), ( , ) 0
ˆ ( , )( , )
0 , ( , ) 0
G k lH k l
H k lF k l
H k l
=
26
C. Nikou – Digital Image Processing (E12)
Restoration in Absence of Noise
The Pseudo-inverse Filter (cont...)
A pseudo-inverse filter also arises by the
unconstrained least squares approach.
Find the image f, that, when it is blurred by
H, it will provide an observation as close as
possible to g, i.e. It minimizes the distance
between Hf and g.
27
C. Nikou – Digital Image Processing (E12)
Restoration in Absence of Noise
The Pseudo-inverse Filter (cont...)
This distance is expressed by the norm:
min ( )Jf
f 20
Hf - g
f
2 0T H Hf -g 2 2T T H Hf H g
1
T T
f H H H g
0J
f
2( )J f Hf - g
28
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
The Inverse Filter
Recall the imaging model with spatially
invariant degradation and noise
g = Hf + η
( , ) ( , ) ( , ) ( , )g x y h x y f x y x y
( , ) ( , ) ( , ) ( , )G k l H k l F k l N k l
29
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
The Inverse Filter (cont...)
Applying the inverse filter in the Fourier
domain:
Even if we know H(k,l) we cannot recover F(k,l)
due to the second term.
If H(k,l) has small values the second term
dominates (it goes to infinity if H(k,l)=0!).
( , ) ( , ) ( , ) ( , )G k l H k l F k l N k l
( , )ˆ ( , ) ( , ) ( , )
N k lF k l F k l
H k l
30
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
The Inverse Filter (cont...)
• One approach to get around the problem is
to limit the ratio G(k,l) / H(k,l) to
frequencies near the origin that have lower
probability of being zero.
• We know that H(0,0) is usually the highest
value of the DFT.
• Thus, by limiting the analysis to
frequencies near the origin we reduce the
probability of encountering zero values.
31
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
The Inverse Filter (cont...)
Blurring degradation
32
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
The Inverse Filter (cont...)Inverse filter with cut-off
33
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
Wiener Filter
• So far we assumed nothing about the statistical
properties of the image and noise.
• We now consider image and noise as random
variables and the objective is to find an estimate
of the uncorrupted image f such that the mean
square error between the estimate and the image
is minimized:
where E[x] is the expected value of vector x.
2
ˆ
ˆmin ( )E f
f f
34
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
Wiener Filter (cont...)
• Recall also the definition of the correlation
matrix between two vectors x and y:
• We assume that the image and the noise
are uncorrelated:
1 1 1 2 1
1 2
N
T
N N N N
E x y E x y E x y
E
E x y E x y E x y
xyR xy
ηf fηR R 0
35
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
Wiener Filter (cont...)
• We are looking for the best estimate:
• Let’s confine our estimate to be obtainable
by a linear operator on the observation:
and the goal is to find the best matrix P.
2
ˆ
ˆmin ( )E f
f f
ˆ f Pg
36
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
Wiener Filter (cont...)
Denoting by the n-th row of P:
2 22ˆ ˆ ˆ( ) ( )J E E E
f f f f f f Pg
2 2ˆ( ) T T
n n n n
n n
J E E f f p g f p g
T
np
37
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
Wiener Filter (cont...)
ˆ( )T
T T
n n n n
n
J E f f p g f p g
T T T T T T
n n n n n n n n
n
E f f p gf f g p p gg p
T T T T T T
n n n n n n n n
n
E E E E f f p gf f g p p gg p
2n n n
T T
n n n
n
f f gf ggR p R p R p
38
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
Wiener Filter (cont...)
• We can now minimize the sum with respect
to each term:
2 0n n n
T T
n n n
n
f f gf ggR p R p R p
p
2 2 0n n gf ggR R p
1
nn
gg gfp R R
1
n
T
n
f g ggp R R
39
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
Wiener Filter (cont...)
• Assembling the rows together:
• We have to compute the two matrices:
1 fg ggP R R
( )( )T TE E ggR gg Hf η Hf η
T T T T T TE Hff H Hfη ηf H η η
T T ff fη ηf ηηHR H HR R H R
40
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
Wiener Filter (cont...)
• Assumming noise is uncorrelated with image:
• Also,
• Finally the matrix we are looking for is
( ) ...T T TE E fg ffR fg f Hf η R H
T gg ff ηηR HR H R
1
1 T T
fg gg ff ff ηη
P R R R H HR H R
41
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
Wiener Filter (cont...)
• The estimated uncorrupted image is
which may be also expressed as
• This result is known as the Wiener filter or
the minimum mean square error (MMSE)
filter.
ˆ f Pg 1ˆ T T
ff ff ηη
f R H HR H R g
1
1 1 1ˆ T T
ηη ff ηη
f H R H R H R g
42
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
Wiener Filter (cont...)
• Special cases:
–No blur (H=I, g=f+η):
–No noise (Rηη=0, g=Ηf):
•This is the inverse filter.
–No blur, no noise (H=I, Rηη=0):
•Do nothing on the observation.
1ˆ
ff ff ηη
f R R R g
1ˆ f Η g
ˆ f g
43
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
Wiener Filter (cont...)
• The size of the matrix to be inverted poses
difficulties and Wiener filter is implemented in the
Fourier domain.
• This occurs when H is doubly block circulant
(represents convolution) and the image f and
noise η are wide-sense stationary (w.s.s).
Definition of a w.s.s. signal:
1) E[f (m,n)]=μ, independent of m,n.
2) E[f(m,n) f(k,l)]=r(m-k,n-l), independent of location.
44
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
Wiener Filter (cont...)
Reminder: the inverse DFT complex exponential
matrix diagonalizes any circulant matrix:
The columns of W-1 are the eigenvectors of any
circulant matrix H.
The corresponding eigenvalues are the DFT values
of the signal producing the circulant matrix.
Remember also that
and that
1 HH W Λ W
T W W
1 1T
W W
45
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
Wiener Filter (cont...)
We will employ the following relations:
If H is real:
1 HH W Λ W
1 1T
T H H
H W Λ W WΛ W
* * *
1 * * 1T T H H
H H WΛ W W Λ W
1 * 1 *1( )N
N
H HW Λ W W Λ W
46
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
Wiener Filter (cont...)
The Wiener solution is now transformed to
the Fourier domain:
Notice that the matrices are diagonal.
1ˆ T T
ff ff ηη
f R H HR H R g
11 1 * 1 1 1 * 1( )( ) ( )( )( ) ( )
ff H H ff H ηη
W Λ W W Λ W W Λ W W Λ W W Λ W W Λ W g
11 * 1 *( )
ff H H ff H ηη
W Λ Λ W W Λ Λ Λ Λ W g
* * 1ˆ ( ) ff H H ff H ηηWf Λ Λ Λ Λ Λ Λ Wg
1 * * 1( ) ff H H ff H ηηW Λ Λ Λ Λ Λ Λ Wg
47
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
Wiener Filter (cont...)
Sff (k,l)=DFT(Rff (m,n)) is the power spectrum
of the image f (m,n).
Sηη(k,l)=DFT(Rηη (m,n)) is the power spectrum
of the noise η(m,n).
* * 1ˆ ( ) ff H H ff H ηηWf Λ Λ Λ Λ Λ Λ Wg
*
2
( , ) ( , )( , ) ( , )
( , ) ( , ) ( , )
ff
ff
S k l H k lF k l G k l
S k l H k l S k l
48
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
Wiener Filter (cont...)
If Sff (k,l) is not zero we may define the Signal
to Noise Ratio in the frequency domain:
and the Wiener filter becomes:
( , )SNR( , )
( , )
ffS k lk l
S k l
*
2 -1
( , )( , ) ( , )
( , ) SNR ( , )
H k lF k l G k l
H k l k l
49
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
Wiener Filter (cont...)
A well known estimate of Sff (k,l) is the
periodogram (the ML estimate of Rff when f is
assumed Gaussian):
In practice, as F(k,l) is unknown, we use
21( , ) ( , )ffS k l F k l
MN
21ˆ ( , ) ( , )ffS k l G k lMN
50
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
Wiener Filter (cont...)
51
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
Wiener Filter (cont...)Inverse WienerDegraded image
Noise variance one order
of magnitude less.
Noise variance ten orders
of magnitude less.
52
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
Constrained Least Squares Filter
• When we do not have information on the power
spectra the Wiener filter is not optimal.
• Another idea is to introduce a smoothness term in
our criterion.
• We define smoothness by the quantity
– Q is a high pass filter operator, e.g. the Laplacian,
– Q is a doubly block circulant matrix representing
convolution.
• We look for smooth solutions minimizing
2Qf
2Qf
53
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
Constrained Least Squares Filter (cont...)
We have the following constrained least
squares (CLS) optimization problem:
Minimize subject to
yielding the Lagrange multiplier minimization of
the function:
2 2( , )J f Hf g Qf
Smoothness termData fidelity term
2Qf Hf g
54
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
Constrained Least Squares Filter (cont...)
2 ( ) 2 0T T H Hf g Q Qf( , ) 0J
ff
1
T T T
f H H Q Q H g
Parameter λ controls the degree of smothness:
1
0, T T
f H H H g pseudo-inverse, ultra rough solution
, 0 f ultra smooth solution
55
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
Constrained Least Squares Filter (cont...)
In the Fourier domain, the constrained least
squares filter becomes:
Keep always in mind to zero-pad the images
properly.
*
2 2
( , )( , ) ( , )
( , ) ( , )
H k lF k l G k l
H k l Q k l
56
C. Nikou – Digital Image Processing (E12)
Restoration in Presence of Noise
Constrained Least Squares Filter (cont...)
Low noise: Wiener and CLS generate equal results.
High noise: CLS outperforms Wiener if λ is properly selected.
It is easier to select the scalar value for λ than to approximate the SNR
which is seldom constant.
57
C. Nikou – Digital Image Processing (E12)
Restoration Performance Measures
Mean square error (MSE):
1 1 2
0 0
1 ˆMSE ( , ) ( , )M N
m n
f m n f m nMN
Original image f and restored image f̂
58
C. Nikou – Digital Image Processing (E12)
Restoration Performance Measures
(cont...)
The Signal to Noise Ratio (SNR) considers the
difference between the two images as noise:
1 12
0 0
1 1 2
0 0
ˆ ( , )
SNRˆ( , ) ( , )
M N
m n
M N
m n
f m n
f m n f m n