super resolution

Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR

Super Resolution

Federico D’Amato Roberto Medico

University of Florence

June 9, 2014

F. D’Amato, R. Medico Super Resolution


Outline

1 Introduction

2 Image Registration

3 Imaging Process

4 IBP

5 Irani and Peleg Algorithm

6 Gradient-like Method

7 Color SR



Super Resolution Techniques

Super Resolution is a class of techniques that enhance theresolution of an imaging system. There are 3 main approachesto SR reconstruction of an high-resolution image from lowerresolution image(s):

• Interpolation-based• Example-learning-based• Multi-image-based



Interpolation-based

Figure: Interpolaton methods try to achieve a best approximation of apixel’s color and intensity based on the values at surrounding pixels



Example-learning-based

Correspondences between low-resolution and high-resolutionimages are learned from a set of training images. The trainingset consists of high-resolution / low-resolution pairs.



Multi-Image Super Resolution

Super-Resolution from image sequences attempts toreconstruct the original scene image with high resolution givena set of observed images at lower resolution.



Why Super Resolution?

Limit of camera resolution:• Spatial limit→ determined by spatial density of optical

sensor• Optical blur→ determined by the lens

How to improve camera resolution?• Direct method: improving imaging system by

manufacturing technique (pixel density, lens size)• Use of Super-resolution reconstruction:

• Use of spatial sub-pixel movement information betweenframe

• Reconstruction from low-resolution image sequences tohigh-resolution image



Digital Imaging System

Key components:1 the sensor⇒ limit on highest spatial frequency2 the lens⇒ optical blur

Figure: Image acquisition process



Spatial Aliasing

Spatial aliasing is an effect that causes different signals tobecome indistinguishable (or aliases of one another) whenspatially-sampled. When a digital image is recorded, areconstruction is performed by the imaging device→ if theimage data is not properly processed during sampling orreconstruction, the reconstructed image will differ from theoriginal image (it’s called an ’alias’ of the original scene)

Figure: One signal and its alias



Aliasing components

• Sensor is constructed from a finite number of discretepixels→ reconstruction of real world scene is affected byaliasing effects

• It’s impossible to completely remove aliasing componentsusing anti-aliasing filters⇒ information in the aliasedcomponents is used to recover spatial frequenciesbeyond sensor resolution

• It’s the possible to use information to improve the imageresolution



Aliasing effect on patterns of increasing frequency→ poor (orcompletely wrong) image reconstruction



Figure: Sub-Pixel shifted signals



Naive approach

• How can we compute the valueof pixel X?

• By applying some interpolationtechnique (e.g. bilinear) toneighbours A,B,C,D of X



Multi-image Approach

• LR image resolution: MxN• Images displacement: half a

pixel• Combining the pixel of the LR

images in a more dense grid2Mx2N returns an image athigher resolution.



Outline

1 Introduction


3 Imaging Process

4 IBP



7 Color SR



Registration

• Computation of the changes (displacements) betweenimages is known as registration

• 2D Rotation matrix:

• Displacement are computed between one image g0 (takenas reference image) and all the others image.Displacement between gk and g0 can be written as:

g0(x , y) = gk (x cos(Θ)−y sin(Θ)+a, y cos(Θ)+x sin(Θ)+b)



Figure: Example of rigid registration



• Expand sin Θ and cos Θ to the first two terms of their Taylorseries:

g0(x , y) ≈ gk (x + a− yΘ− xΘ2

2, y + b + xΘ− y

Θ2

2)

• Expand gk to the first term of its Taylor series:

g0(x , y) ≈ gk (x , y)+(a−yΘ−xΘ2

2)∂gk

∂x+(b+xΘ−y

Θ2

2)∂gk

∂y

• The error function between gk and g0 is:

E(a,b,Θ) =∑

[gk (x , y) + (a− yΘ− xΘ2

2)∂gk

∂x+

+(b + xΘ− yΘ2

2)∂gk

∂y− g0(x , y)]2



• ∂E∂a = 0, ∂E

∂b = 0, ∂E∂Θ = 0

• Ignoring non-linear terms and small coefficients we get thefollowing system of linear equations, whose solution(a,b,Θ) minimizes the difference between g0 and gkwarped by (a,b,Θ):∑

g2x a +

∑gxgyb +

∑Agx Θ =

∑gxgt∑

g2y b +

∑gxgya +

∑Agy Θ =

∑gygt∑

A2Θ +∑

Agyb +∑

Agxa =∑

Agt

where gx = ∂gk∂x , gy = ∂gk

∂y , gt = g0 − gk and A = xgy − ygx



Iterative refinement

When it’s not possibile to assume that the displacementsbetween g0 and gk are sufficiently small, an iterativerefinement algorithm is used:

1 Assume no motion between frames2 for n=0,1,..

• Compute (a(n),b(n),Θ(n)) and add the computed motionto the current estimate (a,b,Θ)

• Warp frame gk towards g0 using (a,b,Θ) and return to 2.

The process ends when (a(n),b(n),Θ(n)) ≈ 0.



Outline

1 Introduction


3 Imaging Process

4 IBP



7 Color SR



Receptive Field

The receptive field of a LR pixel (m,n) of the kth LR image isdefined by its center (x , y) and its shape, determined by theregion of support of hPSF (·) in the high resolution grid. Thecenter (x , y) can be computed by:

x = ak + sxm cos Θk − syn sin Θk

y = bk + sxm sin Θk + syn cos Θk

where• (ak ,bk ) is the translation of the kth image from g0

• Θk is the rotation between the kth image and g0

• sx and sy are the upscaling factors in x and y directions



Figure: Receptive Field



• Imaging process can be modeled as:

gk (m,n) = σk (hPSF (f (x , y)) + ηk (x , y))

where• gk is the kth observed LR image• f is the original image• hPSF is a blurring operator• ηk is an additive noise term• σk is a non-linear function that digitizes and quantizes

image into pixels (including displacement)• (x , y) is the center of the receptive field of the detector

whose output is gk (m,n)



Figure: Simulated Imaging Process



Blurring Operator Estimate

Given a generic imaging device, we can empyrically estimateits blurring function h(·) analyzing the output of the imagingprocess of well-known sample scenes.



Outline

1 Introduction


3 Imaging Process

4 IBP



7 Color SR



Iterative Back Projection

• Iterative algorithm based on a set of K low resolutionimages of the same scene with known displacements

• Goal: to improve an initial guess of the HR imageiteratively minimizing an error function usingback-projection



Hypothesis

• Assumptions:• displacement between images can be described by three

parameters:• a, horizontal shift• b, vertical shift• Θ, rotation angle

• ignores acceleration of the camera while imaging a singleframe



Data:• f 0: initial guess of the HR image• gk : set of LR observed images• hPSF , (ak ,bk ,Θk ) ∀k = 1, ..,K

for n = 0,1, .. do

1 Compute the set of K simulated LR images {g(n)k } from f (n)

2 Compute en between gk and gk(n)

if en > ε thenUpdate guess f (n+1) by back-projecting the error on f (n)

elsereturn f (n)

endend

Algorithm 1: Iterated Back Projection



Outline

1 Introduction


3 Imaging Process

4 IBP



7 Color SR



Simulated Imaging Process

How can we programmatically simulate the device imagingprocess?

Def.A low resolution pixel y is influenced by a high resolution pixelx if x ∈ y’s receptive field

Def.A low resolution image g is influenced by a high resolutionpixel x if ∃y ∈ g influenced by x



g(n)(~y) =∑~x

f (n)(~x)hPSF (~x − ~z~y )

where• hPSF is the point-spread kernel of the imaging blur• ~x is an HR pixel• ~y is a LR pixel influenced by ~x• ~z is the center of y ’s the receptive field



Idea

• Given the g(n)k simulated LR images, the goal is to

minimize the error between {g(n)k } and {gk }.

• The minimization is obtained with the iterativeback-projection scheme, where ek = gk − g(n)

k isweighted computing the influence of every LR pixel ~y onHR pixel ~x , using:

hBP(~x − ~z~y )∑~y∈

⋃k Yk,~x

hBP(~x − ~z~y )

where Yk ,~x is the set: {~y ∈ gk | ~y is influenced by ~x}• ~y has more influence when ~x is close to ~z~y , center of ~y ’s

receptive field

• The error is then multiplied by a factor:hBP(~x−~z~y )

c



Guess Improvement

• The value of ~x in the next guess f (n+1) is calculatedsumming up the weighted errors on all the LR pixel ~y itinfluences

f (n+1)(~x) = f (n)(~x)+∑

~y∈⋃

k Yk,~x

(gk (~y)−g(n)k (~y))

(hBP(~x − ~z~y ))2

c∑

~y∈⋃

k Yk,~xhBP~x~y



Back-Projection Kernel

• hBP~x~y affects how much the error on LR pixel ~y (influenced

by ~x) contributes to the value of HR pixel ~x in the nextguess f (n+1)

• hBP affects the characteristics of the solution image, e.g.its smoothness

• A possible choice is hBP = hPSF



Initial Guess Choice

• Initial guess f (0) can influence the output of the algorithm,i.e. which HR image is reached first

• One possibile choice of f (0) is taking the average of theupscaled LR images gk

• This choice doesn’t affect performance



Algorithm Complexity

The complexity is O(KN min{M, log N}) where:• K is the number of LR images• N is the size of the HR image f• M is the size of HPSF kernel

Parallelism can be used to compute the contributions of LRpixels indipendently.



Improvements

• Stable Pixels: HR pixels that don’t change value for 2consecutive iterations won’t be considered in the followingiterations

• Noise reduction: minimal and maximal values ofgk (~y)− g(n)

k (~y) are ignored in computing the weightedaverage of the contributions of the LR pixels in the iterativeback-projection scheme



Outline

1 Introduction


3 Imaging Process

4 IBP



7 Color SR



Error Function

The error function to minimize is given by the MSE between thesimulated images g(n)

k and the observed images gk :

e(n) =

∑k (gk − g(n)

k )2

K



Algorithm

f (n+1) = f (n) − λG

where• G =

∑k HBP(g(n)

k − gk )

• g(n)k is the kth simulated LR image at the nth iteration



Outline

1 Introduction


3 Imaging Process

4 IBP



7 Color SR



YIQ representation

• Y component represents the luminance information• I and Q represent the chrominance information

Most of the energy is concentrated in the Y component.



Color Super Resolution

It’s possible to use the SR algorithm even on color images,going through 4 steps:

1 transform the color images in YIQ representation2 apply the SR algorithm to the Y component images3 register the images at the two (I,Q) chrominance image

sequences using parameters found in 2. Create anaverage for each of the I and Q components

4 fuse the HR Y component and LR I and Q components togenerate a HR RGB image



References

• ’Image sequence enhancement using sub-pixeldisplacements’, Keren, D., Peleg, S. ; Brada, R.; Dept. ofComput. Sci., Hebrew Univ., Jerusalem, Israel

• ’Video Super-resolution Reconstruction Based onSub-pixel Registration and Iterative Back Projection’,Journal of Electronic Imaging,Vol. 18, No. 1, 2009,Feng-Qing Qin, Xiao-HaiHe, Wei-Long Chen, Xiao-MinYang, and Wei Wu

• ’Improving resolution by image registration’, Michal Irani,Shmuel Peleg

• ’Super-Resolution’, Pradeep Gaidhani


super resolution

Technology