introduction to image super-resolution - department of computer

Introduction to Image Super-resolution

Presenter: Kevin Su

References

1. S.C. Park, M.K. Park, and M.G. KANG, “Super-Resolution Image Reconstruction: A Technical Overview”, IEEE Signal Processing Magazine, Vol. 20, pp. 21-36, May 2003

2. W.T. Freeman, T.R. Jones, and E.C. Pasztor, “Example-Based Super-Resolution”, IEEE Computer Graphics and Applications, Vol. 22, pp. 56-65, 2002.

Overview

• Introduction• Super-resolution Techniques

– Multi-frame Super-resolution– Single-frame Super-resolution

• Conclusion

Terminology

• Low-Resolution (LR):– Pixel density within an image is small, therefore

offering less details.• High-Resolution (HR):

– Pixel density within an image is larger, therefore offering more details.

• Superresolution (SR):– Obtaining a HR image from one or multiple LR

images .

Application

• Medical imaging (ie. CAT, MRI, etc).• Satellite imaging• Enlarging consumer photographs• Video surveillance (ie. Car wash

kidnapping).• Converting NTSC video content to high-

definition television

Application

raw After apply super-resolution technique

Application

zoom apply super-resolution technique

Application

Video with low resolution

Video with high resolution

Application

How to increase resolution?

• Possible ways for increasing an image resolution:

– Reducing pixel size.– Increase the chip-size.– Super-resolution.


• Reduce pixel size:– Increase the number of pixels per unit

area.– Advantage:

• Increases spatial resolution.– Disadvantage:

• Noise introduced.• As the pixel size decreases, the amount of

light decreases.


• Increase the chip size (HW):

– Advantage:• Enhances spatial resolution.

– Disadvantage:• High cost for high precision optics.


• Superresolution (SR):– Process of combining multiple low

resolution images to form a high resolution image.

– Advantages:• Cost less than comparable approaches.• LR imaging systems can still be utilized.

Overview



• Conclusion

Multi-frame Super-resolution • How can we obtain a HR image from multiple

LR images?– Basic premise is the availability of multiple LR

image captured form the same scene.– These multiple LR images provide different “looks”

at the same scene. – Each LR is naturally shifted with subpixel precision.– If LR images are shifted by integer units, then each

image contains the same information, SR is not possible.

– If LR images have different subpixel shifts, then SR is possible.

Basic premise for SR

Common image acquisition System

Observation ModelFirst step to understanding SR is to formulate an Observation Model to relate the LR images to the desired HR image.

• Desired HR image:– Size:– Vector: where

• LR images:– Size: – K-th LR image:

where

Observation Model

2211 NLNLN ×=2211 NLNL ×

Tk,Mk,2k,1 ,....,y,yy ][yk =

21 NN ×

TNxxxx ],....,,[ 21=

21...,21 NNMandp,,k ×==

• Observation model can be represented as follows:

– is a warp matrix– represents blur matrix– is a sub-sampling matrix– represents noise matrix

• Without loss of generality, it can also be represented as follows:

Observation Model

DkBkM

pkfornxMDB kkk ≤≤+= 1yk

pkfornxW kk ≤≤+= 1yk

kn

Nonuniform interpolation approach

• 3 stages: – Registration– Interpolation– Deblurring


• Registration:– Need to estimate the scene motion for each image

with reference to one particular image.– The motion can be estimated as a 1-to-1

representation between the reference image and each of the images.


• Registration:– Estimating the completely arbitrary motion in

real world image scenes is extremely difficult, with almost no guarantees of estimator performance.

– Incorrect estimates of motion have disastrous implications on overall SR performance.


• Interpolation:– Since the shifts between the LR images are

arbitrary, the images will not always match up to a uniformly to the HR grid.

– Thus, nonuniform interpolation is necessary to obtain a uniformly spaced HR image from a nonuniformly spaced composite of LR images.

– Nonuniform interpolation between LR images are used to improve resolution.


• Interpolation:– This step requires interpolation when the estimated

fractional unit of motion is not equal to the HR grid reference image.


• Deblurring:– In SR, blur is usually modeled as a spatial

averaging operator as shown below.

Result

Regularized SR Reconstruction

• If there are enough LR images, we can solve

• In reality, it is hard to find sufficient number of LR images. Use procedure (called regularization) to stabilize the inversion of ill-posed problem.

– Deterministic Approach (CLS)

– Stochastic Approach (MAP)

pkfornxW kk ≤≤+= 1yk

Deterministic Approach (CLS)• CLS can be formulated by choosing x to

minimize the Lagrangian

– C is generally a high-pass filter– α is regularization parameter

• The cost function above is convex and differentiable with the use of a quadratic regularization term. We can find a unique estimate image using iterative techniques

⎥⎦

⎤⎢⎣

⎡+−∑

=

p

kkk CxxWy

1

2 α

x̂k

p

k

Tk

p

k

Tk

Tk yWxCCWW ∑∑

==

=⎥⎦

⎤⎢⎣

⎡+

11

ˆα

Stochastic Approach (MAP)

• Bayesian approach provides a flexible and convenient way to model a priori knowledge concerning solution

• Using MRF Gibbs priori to define P(x)

).,...,,|(maxarg 21 pyyyxPx =

)}.(ln)|,...,,(max{lnarg 21 xPxyyyPx p +=

∑∈

−=−==Sc

c xZ

xUZ

xXP ))(exp(1)}(exp{1)( ϕ

Result

Other Approaches

• Frequency Domain Approach• Projection onto Convex Sets Approach• ML (maximum likelihood approach)• ML-POCS hybrid approach• Iterative back-projection approach• Adaptive Filtering Approach• Motionless SR Reconstruction

Approach

Overview



• Conclusion

Single-frame SR• Traditional resolution enhancement:

– Smoothing (Gaussian, Wiener, and median filters)– Interpolation (Nearest ngbr, bilinear, bicubic and

cubic spline etc)– Sharpening by amplifying existing image details

(it is useful to do, provided noise isn’t amplified)• Single-frame SR:

– Estimate missing high-resolution detail that isn’t present in the original image, and which we can’t make visible by simple sharpening

Example-based SR

• Algorithm uses a training set to learn the fine details of an image at low-resolution.

• It then uses those learned relationships to predict fine details in other images.– Markov network– One pass algorithm

Training Set Generation• Start with a collection of HR images.• For each HR image, degrade it to get a LR

image.– Blur & subsample each to create LR image of ¼ total

pixels.• Apply analytical interpolation to the LR image.

– ie. Cubic spline.– This will generate an image of desired # of pixels, but

lacking the HR detail.• Band pass filter and contrast normalize the

interpolated image AND the original HR image.

Training Set Generation

Training Set Generation• Divide images into small patches:

– 5x5 (HR), 7x7 (LR)

Markov network

Markov network

• Select the 16 or so closet examples to each input patch as the different states of the hidden nodes, x, that we seek to estimate.

• Maximize

where

, the sum of squared differences between patch candidates xi and xjin their overlap regions at nodes i and j

∏ ∏=)(

),(),(1)|(ij k

kkkjiij yxxxZ

yxP φψ

⎥⎦

⎤⎢⎣

⎡−= 22

),(exp),(

σψ jiij

jiij

xxdxx

),( jiij xxd

One pass algorithm

Results

Training Set

Results

Conclusions and future works

• Current SR approaches are effective to some extent

• SR considering registration error: – Use total least squares method to minimize the error– Use channel adaptive regularization: SR images with

large registration error should be less contributed to the estimate of the HR.

• Blind SR Image Reconstruction: when blurring process is unknown. Need blur identification.

• Computationally efficient SR Algorithm

introduction to image super-resolution - department of computer

Documents