image denoising using locally learned dictionaries priyam chatterjee peyman milanfar dept. of...

Image Denoising using Locally Learned Dictionaries

Priyam ChatterjeePeyman Milanfar

Dept. of Electrical EngineeringUniversity of California, Santa Cruz

Computational Imaging VII – 20 Jan, 2009

20 Jan, 2009 Image Denoising using Locally Learned Dictionaries 2

Overview

Data Model

Kernel Regression for Denoising

Denoising with Locally Learned Dictionaries (K-LLD)

Results

Conclusions

20 Jan, 2009 Image Denoising using Locally Learned Dictionaries 3

Data Model

Pointwise data model

Patchwise model

Locally smooth functionto be estimated

Zero-mean I.I.D. noise

Observation

denoted as

20 Jan, 2009 Image Denoising using Locally Learned Dictionaries 4

Steering Kernel Regression (SKR)

Optimization problem

Solution: Nonlinear filters Polynomial basis

Data-dependent

weights

20 Jan, 2009 Image Denoising using Locally Learned Dictionaries 5

SKR Weights

Weights based on pixel “self-similarity” in a local patch

Covariance matrix takes into account: orientation and strength of edges

Gradient Covariance

20 Jan, 2009 Image Denoising using Locally Learned Dictionaries 7

SKR Weights

Note how the weights adapt to the underlying image structure

Noisy Noise-free

H. Takeda, S. Farsiu, and P. Milanfar, “Kernel Regression for Image Processing and Reconstruction”, IEEE Trans. on Image Processing, vol. 16, no. 2, pp. 349-366, February 2007.

20 Jan, 2009 Image Denoising using Locally Learned Dictionaries 8

Now we extend it …..

is fixed order, everywhere -- not depending on underlying image structure

• Lower orders fit flat regions, higher order for texture and fine details

Global dictionary does not adapt to local image characteristics

• Dictionary atoms should capture underlying local image structure

20 Jan, 2009 Image Denoising using Locally Learned Dictionaries 9

Denoising with Locally Learned Dictionaries (K-LLD)

Identify dictionary which best captures underlying geometric structure

Similar structures will have similar dictionary, similar weights

Cluster image based on geometric similarity (K-Means on the SKR weights)

Learn dictionary and order of regression for each cluster

20 Jan, 2009 Image Denoising using Locally Learned Dictionaries 10

K-LLD: Algorithm Outline

Calculate

weights

Learn

dictionaries

Clustering

Iter

ate

Noisy Image

Kernel Regression

Denoised Image

20 Jan, 2009 Image Denoising using Locally Learned Dictionaries 11

Class 1Class 1

Class KClass KClustering StageClustering Stage

K-LLD : Algorithm Outline

Dictionary Selection StageDictionary Selection Stage

Noisy ImgCalculate Steering

Weights

Calculate SteeringWeights

Coefficient Calculation Stage

Coefficient Calculation StageDenoised

Img

20 Jan, 2009 Image Denoising using Locally Learned Dictionaries 12

SegmentImage

SegmentImage

Clustering Stage

K-MeansK-MeansClass 1Class 1

Class KClass K

K-LLD : Algorithm Outline

Dictionary Selection StageDictionary Selection Stage

Noisy ImgCalculate Steering

Weights

Calculate SteeringWeights

Coefficient Calculation Stage

Coefficient Calculation StageDenoised

Img

20 Jan, 2009 Image Denoising using Locally Learned Dictionaries 13

Clustering Stage

Objective : Cluster image based on geometric similarity of underlying data

• Feature Selection

• What features capture data geometry ?

• Distance Metric

• What metric captures distance between features ?

• Clustering Algorithm

• What algorithm segments the image best ?

20 Jan, 2009 Image Denoising using Locally Learned Dictionaries 14

K-Means for Clustering

Features : normalized steering wts

Distance Metric : L2

Initialization : Randomly initialize cluster centers

Run K-Means multiple times and select result that minimizes within-cluster distance

20 Jan, 2009 Image Denoising using Locally Learned Dictionaries 15

Noi

se-f

ree

Noi

sy

Clustering the noise-free image

Clustering the noisy image

20 Jan, 2009 Image Denoising using Locally Learned Dictionaries 16

SegmentImage

SegmentImage

Clustering Stage

K-MeansK-MeansClass 1Class 1

Class KClass K

K-LLD : Algorithm Outline

Dictionary Selection StageDictionary Selection Stage

Noisy ImgCalculate Steering

Weights

Calculate SteeringWeights

Coefficient Calculation Stage

Coefficient Calculation StageDenoised

ImgDictionary Selection Stage

PCAPCAFormDictionary

FormDictionary

20 Jan, 2009 Image Denoising using Locally Learned Dictionaries 17

Dictionary Selection

Represent each patch in the kth cluster

Variable Proj.

Solved by PCA

Mean patch of k-th cluster

Enforceorthonormality

20 Jan, 2009 Image Denoising using Locally Learned Dictionaries 18

Dictionary Selection

PCA in each cluster to form a dictionary

Describe data without fitting noise

Number of atoms based on cluster geometry • Clusters with flat regions need fewer atoms, finer

details need more

constant

Singular values

patch sizeNo. of atoms

20 Jan, 2009 Image Denoising using Locally Learned Dictionaries 19

Example : House Image

Dictionary atoms for AWGN of std. dev. 15

Clu

ster

Ato

m 1

Ato

m 2

Ato

m 3

20 Jan, 2009 Image Denoising using Locally Learned Dictionaries 21

Example : Noise-free clusteringC

lust

erA

tom

1A

tom

2A

tom

3A

tom

4

Few of the atoms in the dictionaries for different clusters

Brick facade

20 Jan, 2009 Image Denoising using Locally Learned Dictionaries 23

Algorithm Outline

SegmentImage

SegmentImage

Clustering Stage

Class 1Class 1

Class KClass KK-MeansK-Means

Noisy ImgCalculate Steering

Weights

Calculate SteeringWeights

Coefficient Calculation Stage

Coefficient Calculation Stage

Denoised Img

Dictionary Selection Stage

PCAPCAFormDictionary

FormDictionaryKernel RegressionKernel Regression

Denoised Img

20 Jan, 2009 Image Denoising using Locally Learned Dictionaries 24

Kernel Regression

Weighted least squares solution

Final estimate

Coefficient Calculation

center pixel of patch .

20 Jan, 2009 Image Denoising using Locally Learned Dictionaries 25

Kernel RegressionKernel RegressionDenoised

Img

Algorithm Outline

SegmentImage

SegmentImage

Clustering Stage

Class 1Class 1

Class KClass KK-MeansK-Means

Noisy ImgCalculate Steering

Weights

Calculate SteeringWeights

Dictionary Selection Stage

PCAPCAFormDictionary

FormDictionary

20 Jan, 2009 Image Denoising using Locally Learned Dictionaries 26

Iteration

Re-learn weights (features) from denoised image

Perform clustering of updated image using new features

Learn dictionary from updated image

Kernel regression on input noisy image• Preserves edges and finer structures

20 Jan, 2009 Image Denoising using Locally Learned Dictionaries 28

K-LLD: Algorithm Outline

Calculate

weights

Learn

dictionaries

Clustering

Iter

ate

Noisy Image

Kernel Regression

Denoised Image

Original Noisy Image

Results – AWG noise (std dev 25)

K-LLD, MSE 96.95 SSIM 0.825

BM3D, MSE 88.82 SSIM 0.841Original Parrot Image

K-SVD, MSE 101.54 SSIM 0.826

SKR, MSE 99.96 SSIM 0.826

Noisy Image

20 Jan, 2009 Image Denoising using Locally Learned Dictionaries 31

More ResultsM

SE

SS

IM

Results – Real noise & colorISKR K-LLDBM3D

Color Results

ISKR, Order 2

BM3DOriginal Image

K-LLD

20 Jan, 2009 Image Denoising using Locally Learned Dictionaries 35

Thank you

P. Chatterjee and P. Milanfar, “Clustering-based Denoising with Locally LearnedDictionaries”, Accepted for publication in IEEE Trans. Image Processing

Available at:

http://www.ee.ucsc.edu/~milanfar

20 Jan, 2009 Image Denoising using Locally Learned Dictionaries 36

Iterative Scheme

Why iterate ?

• Weights true to underlying structure in presence of lesser noise

• Better weights means better clustering

• Dictionary captures underlying data better when learned on less noisy image