yao wang - nyu tandon school of engineeringeeweb.poly.edu/~yao/ee3414/image_filtering.pdf · ©yao...

41
Yao Wang Polytechnic University, Brooklyn, NY11201 http://eeweb.poly.edu/~yao Image Filtering: Noise Removal, Sharpening, Deblurring

Upload: lamkhanh

Post on 30-Jan-2018

220 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

Yao WangPolytechnic University, Brooklyn, NY11201

http://eeweb.poly.edu/~yao

Image Filtering: Noise Removal, Sharpening,

Deblurring

Page 2: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 2

Outline

• Noise removal by averaging filter• Noise removal by median filter• Sharpening (Edge enhancement)• Deblurring

Page 3: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 3

Noise Removal (Image Smoothing)

• An image may be “dirty” (with dots, speckles,stains)• Noise removal:

– To remove speckles/dots on an image– Dots can be modeled as impulses (salt-and-pepper or

speckle) or continuously varying (Gaussian noise)– Can be removed by taking mean or median values of

neighboring pixels (e.g. 3x3 window)– Equivalent to low-pass filtering

• Problem with low-pass filtering– May blur edges– More advanced techniques: adaptive, edge preserving

Page 4: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 4

Example

Page 5: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 5

Averaging Filter

• Replace each pixel by the average of pixels in a square window surrounding this pixel

• Trade-off between noise removal and detail preserving:– Larger window -> can remove noise more effectively, but

also blur the details/edges

Page 6: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 6

Example: 3x3 average

100100100100100

100195205200100

100200200195100

100203205200100

100100100100100

100100100100100

100144166144100

100168200167100

100145167144100

100100100100100

Page 7: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 7

Example

Page 8: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 8

Weighted Averaging Filter

• Instead of averaging all the pixel values in the window, give the closer-by pixels higher weighting, and far-away pixels lower weighting.

• This type of operation for arbitrary weighting matrices is generally called “2-D convolution or filtering”. When all the weights are positive, it corresponds to weighted average.

• Weighted average filter retains low frequency and suppresses high frequency = low-pass filter

∑∑−=−=

−−=L

Lk

L

Lllnkmslkhnmg ),(),(),(

Page 9: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 9

Graphical Illustration

Page 10: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 10

Example Weighting Mask

111

111

111

121

242

121

×91

×161

All weights must sum to one

Page 11: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 11

Example: Weighted Average

100100100100100

100195205200100

100200200195100

100203205200100

100100100100100

100100100100100

100156175156100

100175201174100

100158176156100

100100100100100

121

242

121

×161

100100100100100

100144166144100

100168200167100

100145167144100

100100100100100

111

111

111

×91

Page 12: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 12

Filtering in 1-D: a Review

• Continuous-Time Signal

• Extension to discrete time 1D signals

dtethfH

fHfSfG

dtshthtstg

ftj∫

∞−

∞−

=

=

−=∗=

π

τττ

2)()( :responsefrequency Filter

)()()( :DomainFrequency

)()()()()( :Domain Time

2/ toscorrespond 1/22121 range at thelook toneedsonly periodic, is )(

)()( :(DTFT) responsefrequency Filter )()()( :Domain Frequency

)()()()()( :n)convolutio(linear Domain Time

2

s

fnj

n

m

f)/,/(-ffH

enhfHfHfSfG

mnsmhnhnsns

==

−=∗=

−∑

π

Page 13: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 13

Filtering in 2D

).2/1,2/1(),2/1,2/1( region square at thelook toneedsonly periodic, is ),(

),(),(

Filter 2D theof responseFrequency ),(),(),(domain frequency 2DIn

),(),(),(

nConvolutioLinear 2D averaging Weighted

21

21

)(221

212121

1

0

211

0

1

0

1

0

−∈−∈

=

=

−−=

=

∑∑

∑∑

=

+−

=

==

ffffH

enmhffH

ffHffSffG

lnkmslkhnmg

k

kn

nfmfjl

lm

k

kk

l

ll

π

Page 14: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 14

Frequency Response of Averaging Filters

• Averaging over a 3x3 window

=

0000001110011100111000000

91),( nmh

( )

( )

( )

( ) ( ) ( )

( )( ) ( )( )212222

22222222

))1()1((2))1()0((2))1()1((2

))0()1((2))0()0((2))0()1((2

))1()1((2))1()0((2))1()1((221

2cos212cos219111

91

19111

911

919191

91),(

2211

21111211

212121

212121

212121

ffeeee

eeeeeeee

eee

eee

eeeffH

fjfjfjfj

fjfjfjfjfjfjfjfj

ffjffjffj

ffjffjffj

ffjffjffj

ππππππ

ππππππππ

πππ

πππ

πππ

++=++++=

+++⋅+++++=

+++

+++

++=

−−

−−−−

⋅+⋅−⋅+⋅−⋅+−⋅−

⋅+⋅−⋅+⋅−⋅+−⋅−

−⋅+⋅−−⋅+⋅−−⋅+−⋅−

Page 15: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 15

( ) ( )2211

2121

2cos2131)(,2cos21

31)(

)()(),(

ffHffH

fHfHffH

ππ +=+=

=

Sketch H(f1)

05

1015

2025

0

10

20

30-0.5

0

0.5

1

Page 16: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 16

Frequency Response of Weighted Averaging Filters

[ ]

( )( ) 2

22

2

)1/()2cos(2)2cos(2),(

;111

1

)1(1

11

11

)1(1

bvbubvuH

bbb

bbbb

b

bH

+++=

+=

+=

ππ

05

1015

2025

0

10

20

300

0.2

0.4

0.6

0.8

1

Page 17: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 17

Averaging vs. Weighted Averaging

[ ]

( )( ) 9/)2cos(21)2cos(21),(

;111111

111111111

91

vuvuH

H

ππ ++=

=

= [ ]

( )( ) 2

22

2

)1/()2cos(2)2cos(2),(

;111

1

)1(1

11

11

)1(1

bvbubvuH

bbb

bbbb

b

bH

+++=

+=

+=

ππ

05

1015

2025

0

10

20

300

0.2

0.4

0.6

0.8

1

05

1015

2025

0

10

20

30-0.5

0

0.5

1

b=2

Page 18: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 18

Interpretation in Freq Domain

Low-passed image spectrum

Noise spectrum

Filter response

f0

Original image spectrum

Noise typically spans entire frequency range, where as natural images have predominantly lower frequency components

Page 19: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 19

Median Filter

• Problem with Averaging Filter – Blur edges and details in an image– Not effective for impulse noise (Salt-and-pepper)

• Median filter:– Taking the median value instead of the average or weighted

average of pixels in the window• Median: sort all the pixels in an increasing order, take the middle one

– The window shape does not need to be a square– Special shapes can preserve line structures

• Order-statistics filter– Instead of taking the mean, rank all pixel values in the window, take

the n-th order value.– E.g. max or min

Page 20: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 20

Example: 3x3 Median

100100100100100

100195205200100

100200200195100

100203205200100

100100100100100

100100100100100

100100195100100

100200200200100

100100200100100

100100100100100

Matlab command: medfilt2(A,[3 3])

Page 21: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 21

Example

Page 22: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 22

Original Image Corrupted Image Filtered Image

Matlab Demo: nrfiltdemo

Can choose between mean, median and adaptive (Wiener) filter with different window size

Page 23: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 23

Noise Removal by Averaging Multiple Images

Page 24: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 24

Image Sharpening

• Sharpening : to enhance line structures or other details in an image

• Enhanced image = original image + scaled version of the line structures and edges in the image

• Line structures and edges can be obtained by applying a difference operator (=high pass filter) on the image

• Combined operation is still a weighted averaging operation, but some weights can be negative, and the sum=1.

• In frequency domain, the filter has the “high-emphasis” character

Page 25: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 25

Frequency Domain Interpretation

sharpened image spectrum

Filter response (high emphasis)

f0

Original image spectrum

Page 26: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 26

Highpass Filters

• Spatial operation: taking difference between current and averaging (weighted averaging) of nearby pixels– Can be interpreted as weighted averaging = linear convolution– Can be used for edge detection

• Example filters

– All coefficients sum to 0!

;111181111

;111181111

;010141

010;

010141010

−−−−−−−−

−−−

Page 27: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 27

Example Highpass Filters

05 10

1520 25

0

10

20

300

2

4

6

8

05

1015

2025

0

10

20

300

2

4

6

8

10

12

−−−

010141

010

−−−−−−−−

111181111

Page 28: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 28

Example of Highpass Filtering

Original image Isotropic edge detection Binary image

Page 29: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 29

Designing Sharpening Filter Using High Pass Filters

• The desired image is the original plus an appropriately scaled high-passed image

• Sharpening filterhs fff λ+=

),(),(),( nmhnmnmh hs λδ +=

x

f(x)

x

fs(x)=f(x)+ag(x)

x

g(x)=f(x)*hh(x)

Page 30: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 30

Example Sharpening Filters

x

f(x)

x

fs(x)=f(x)+ag(x)

x

g(x)=f(x)*hh(x)

.1010181

010

41

010141

010

41 =

−−−

−=⇒

−−−

−= λwithHH sh

.11111161111

81

111181111

81 =

−−−−−−−−

=⇒

−−−−−−−−

= λwithHH sh

Page 31: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 31

Example of Sharpening

−−−−−−−−

=1111161111

81

sH

−−−−−−−−

=111181111

41

hH

Page 32: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 32

Example of Sharpening

4=λ

8=λ

Page 33: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 33

Challenges of Noise Removal and Image Sharpening

• How to smooth the noise without blurring the details too much?

• How to enhance edges without amplifying noise?• Still a active research area

Page 34: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 34

Wavelet-Domain Filtering

Courtesy of Ivan Selesnick

Page 35: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 35

Feature Enhancement by Subtraction

Taking an image without injecting a contrast agent first. Then take the image again after the organ is injected some special contrast agent (which go into the bloodstreams only). Then subtract the two images --- A popular technique in medical imaging

Page 36: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 36

Image Deblurring

• Noise removal considered thus far assumes the image is corrupted by additive noise– Each pixel is corrupted by a noise value, independent of

neighboring pixels• Image blurring

– When the camera moves while taking a picture– Or when the object moves– Each pixel value is the sum of surrounding pixels � The

blurred image is a filtered version of the original• Deblurring methods:

• Inverse filter: can adversely amplify noise• Wiener filter = generalized inverse filter• Many advanced adaptive techniques

Page 37: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 37

Example of Motion Blur

Page 38: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 38

Inverse Filtering vs. Wiener Filtering

Page 39: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 39

Constrained Least Squares Filtering

Page 40: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 40

What Should You Know

• How does averaging and weighted averaging filter works?• How does median filter works?• What method is better for additive Gaussian noise?• What method is better for salt-and-pepper noise?• How does high-pass filtering and sharpening work?• For smoothing and sharpening:

– Can perform spatial filtering using given filters– Deriving frequency response is not required, but should know the

desired shape for the frequency responses in different applications• What is the challenge in noise removal and sharpening?• What causes blurring? • Principle of deblurring: technical details not required

Page 41: Yao Wang - NYU Tandon School of Engineeringeeweb.poly.edu/~yao/EE3414/image_filtering.pdf · ©Yao Wang, 2006 EE3414: Image Filtering 8 Weighted Averaging Filter • Instead of averaging

©Yao Wang, 2006 EE3414: Image Filtering 41

References

Gonzalez and Woods, Digital image processing, 2nd edition, Prentice Hall, 2002. Chap 4 Sec 4.3, 4.4; Chap 5 Sec 5.1 – 5.3 (pages 167-184 and 220-243)