super-resolution - lina...

Super-ResolutionSuper Resolution

l i ( ) i i i hSuper-Resolution (SR) image re-construction is the process of combining the information from multiple Low-Resolution (LR) aliased and noisy frames of theLow Resolution (LR) aliased and noisy frames of the same scene to estimate a High-Resolution (HR) un-aliased and sharp/de-blurred image.

SR Observation ModelSR Observation Model

The values of the pixels in the k-th low-resolution frame Y of the sequence can be

Warping (Fk)resolution frame Yk of the sequence can be expressed in matrix notation as:

Kkkkkkk ,...,1 ,... =+= nzFHDy

Blurring (H )

where yk and z are, respectively, the lexicographic form of Yk and the undegraded HR image Z, nki h ddi i i d D H F h Blurring (Hk)is the additive noise and Dk, Hk, Fk are the downsampling, blurring, & sub-pixel warping matrices. Yk: N1 ×N2 pixels; Z: rN1 x rN2 , where r is the

Downsampling (Dk)magnification factor.

Inverting (Dk.Hk.Fk) to obtain z is not a trivial task especially that the system is ill conditionedtask especially that the system is ill-conditioned

Existing Super-Resolution ApproachesExisting Super Resolution Approaches

Iterative SR methodsProjection onto convex sets (POCS) [Stark et al 1989]- Projection onto convex sets (POCS) [Stark et al., 1989]

- Least-square error minimization [Schultz et al., 1996]- Regularized maximum a Posteriori (MAP) methods- Regularized maximum a Posteriori (MAP) methods

[Hardie et al., 1997]Fusion-Restoration (FR) SR methods (also known as two-Fusion Restoration (FR) SR methods (also known as twostep methods) [Elad et al., 2001; Farsiu et al., 2004; Hardie et al., 2007]

- Non-iterative fusion step followed by a restoration step.- More computationally efficient Perceptual-based Super-Resolution (Karam et al., 2011)

MAP-Based SR Algorithm [Hardie et al. 97]

The HR image estimate can be computed by maximizing the a posteriori probability, Pr(X|{Yk}), or by maximizing the log-likelihood function:

This results in the following cost function to be minimized assuming a ( ) Kkk ,...,1 ,)]|log[Pr(maxarg == yzz

g gGaussian distribution for the noise and Pr(z|{yk}):

( )( ) zzWzyWzyz 12 2

12

1)( −+−−= Cf TT

σwhere y is a vector concatenating all the LR observations yk, k=1,..,K, σn isthe noise standard deviation and C is the covariance matrix of z, W is the degradation matrix

22 nσ

degradation matrix.An iterative gradient descent minimization procedure is used to update the HR

estimate as follows:flll zzzzz =+ ∇−= |)(1 ε where εl is the step size

lflll zz=+ |)(1 where εl is the step size

Fast Two-Step (FTS) SR Algorithm [Farsiu04][Farsiu04]Fast Two-Step SR (FTS) algorithm consists of a non-iterative data fusion step followed by an iterative gradient-descent deblurring stepfollowed by an iterative gradient-descent deblurring step. Data Fusion Step: Estimated by registration followed by a median operator resulting in a blurred version of the HR frame, , where is the deblurred HR frame

ZHX ˆ.ˆ = Xdeblurred HR frame.Deblurring Step: Reduces to the problem of estimating from and can be formulated as

X Z

)(.)ˆ(minargˆ ZXHZAZ BTVΓ+−= λis the regularization parameter

A is a diagonal matrix with values equal to the square root of the number of pixels that contribute to generate the data fusion matrix

i bil l l i i l i i i b

)()(g1 BTV

λ

ZΓ is a bilateral total variation regularization term given by:

where the operators shift X by l and m pixels in the horizontal and vertical directions

BTVΓ

10

..)( ZSSZZ my

lx

P

Pl

P

m

mlBTV −=Γ ∑∑

−= =

+α 0≥+ml

ml SSwhere the operators shift X by l and m pixels in the horizontal and vertical directions generating several scales of derivatives. Is a spatially decaying factor

yx SS ,α 10 <<α

Fast Two-Step (FTS) SR Algorithm [Farsiu04] (Cont )[Farsiu04] (Cont.)The solution of the deblurring step can be obtained using an iterative gradient-descent scheme as follows:descent scheme as follows:

{( ) ⎪⎫

+−−=+

∑∑ ˆˆ

)ˆˆ(ˆˆ1

P Pl

nTT

nn XZAHsignAHZZ β

where and is the step size in the direction of the gradient.

( )⎪⎭

⎪⎬⎫

−− −−

−= =

+∑∑ )ˆ..ˆ(..0

nmy

lxn

my

lx

Pl m

ml ZSSZsignSSIαλ

0≥+ml βwhere and is the step size in the direction of the gradient.At each iteration, every pixel on the HR grid is processed through the deblurring stage in original SR methodResults in a relatively high computational complexity.

β

Results in a relatively high computational complexity.

Perceptual SR Framework

StrategyReduce the computational complexity by finding a perceptually significant constraint set of pixels to process while maintaining the desired HR visual quality.q y

Selective Perceptual (SELP) SR: Perceptual contrast sensitivity threshold model used to determine perceptually significant pixels (active pixels)(active pixels).Perceptual Attentive (PA) SR: Visual Attention information of salient regions used to further reduce the set of processed pixels g p pand to processed active pixels in attentive regions with a higher accuracy.

Proposed Perceptual Attentive (PA) SR FrameworkFrameworkA computationally efficient Perceptual Super-Resolution framework that exploits human visual pperception and/or visual saliency to significantly reduce SR computations. The method divides the image into:

Active regions consisting of perceptually significant (“active”) pixels determined by contrast sensitivity threshold model.Attentive active regions determined by active

Bilinearly interpolated LR image (q =4)

Visual Attention regions

Attentive active regions determined by active pixels lying in the visually attended regions.

At each iteration of the SR algorithm, the active pixels on visible edges determined by the contrastpixels on visible edges determined by the contrast sensitivity threshold model and lying inside the attentive region are treated with higher accuracy than the pixels in the background region.

Perceptual Active pixels (Active Background)

VA and Perceptual Active pixels (Active Forground)

Proposed Perceptual Attentive (PA) SR Framework

Let z0 = initial HR estimate

Framework

Determine active pixels using perceptual JND model

Update HR estimate for

No

Determine active pixels in attentive regions using perceptual JND model and Saliency Map

active pixels

Yes

No

Yes

Update HR estimate for active pixels in attentive

regions

Max Iterations Reached?

YesNo

No

YesMax

Iterations Reached?

No

STOP

Perceptual Contrast Sensitivity Threshold ModelModel

The contrast sensitivity threshold is used to detect the set of ll i ifi i l ( i i l )perceptually significant pixels (active pixels).

The contrast sensitivity threshold is the measure of the smallest contrast or Just Noticeable Difference (JND) that yields acontrast, or Just Noticeable Difference (JND), that yields a visible signal over a uniform background.JND is computed per image block based on block meanJND is computed per image block based on block mean luminance (8 by 8 blocks used) and using initial SR estimate (obtained by interpolating one of the LR images or by applying a median shift and add of the LR images)JND thresholds are precomputed for all possible discrete mean l i d t d i l k t blluminances and stored in a look-up table.


• For each 8x8 image block, the mean of the block is computed and the corresponding JND threshold t is retrieved (Look up Table)corresponding JND threshold tJND is retrieved (Look up Table).

• Once the tJND is obtained, the center pixel of a sliding 3 × 3 window is compared to its 4 cardinal neighbors.

• If any absolute difference is greater than the tJND , then the corresponding pixel mask is flagged as ’1’ and the pixel is labeled as ’active’ pixel.

• The window will scan all the pixels of the HR image estimateThe window will scan all the pixels of the HR image estimate.


The luminance-adjusted contrast sensitivity JND thresholds are approximated using a power function [Watson93] as follows:using a power function [Watson93] as follows:

Tblk blk

aN

n

N

nnnI

tt ⎟⎟⎟⎞

⎜⎜⎜⎛∑ ∑

− −

= =0 0,

1

1

1

2

21

The contrast sensitivity threshold t128 is evaluated by mapping a parabolic

blkJND N

tt

⎟⎟⎟⎟

⎠⎜⎜⎜⎜

⎝

=)128(2128 where is set to 0.649Ta

The contrast sensitivity threshold t128 is evaluated by mapping a parabolic approximation of a perceptual model proposed by [Ahumada92] to the spatial domain:

Lmax: maximum luma

minmax128 LL

TMt g

−=

Lmin: minimum lumaMg: # of grayscale levels‘T’ based on the model proposed by [Ahumada92]proposed by [Ahumada92]

Visual Attention (VA)( )

HVS perceives a large field of view by a number of p g yfixation points, attended to with high visual acuity, connected by fast eye movements referred to as saccades.VA models provide a relative quantitative measure of the attended regionsof the attended regions.It is stated that artifacts in salient regions are likely to be more annoying to the observer than artifactsto be more annoying to the observer than artifacts in less salient regions.

Visual AttentionVisual Attention

VA Models produce saliency mapsThe saliency map (SM) is simply a likelihood map in which

regions with large values have higher probability of being selected by the HVS as a fixation region as compared to regions with lowerthe HVS as a fixation region as compared to regions with lower values.

The resulting attended regions are used in the proposed Perceptual Attentive (PA) Super-Resolution method to promote the detected subset of attended edge pixels for further enhancement.

The visual attention model purposed by Itti et al [Itti98 Koch06] isThe visual attention model purposed by Itti et al. [Itti98, Koch06] is used in our implementation

Any other good VA model can be used with the proposed y g p pframework.

Simulation results – Fusion-Restoration SR

256 × 256 high-resolution images are used to produce, each, a simulated sequence of sixteen 64×64 low-resolution images:q g

Blurred with a 4x4 Gaussian filter with std=1 .Shifted by multiples of 0.25 pixels in all different directions.Subsampled by a factor of 4 in each directionSubsampled by a factor of 4 in each direction.An additive Gaussian noise of variance 10 is added to the resulting LR sequence.

Clock-256x256 HR Frame LR Frames Carrier-256x256 HR Frame LR Frames Cameraman-256x256 HR LR Frames

Simulation resultsFTS [Farsiu 04]SELP-FTS [Proposed]Simulation results

(a) Original image (b) Bilinearly interp. LR image with 4 VA regions

(c) FTS method (d) SELP-FTS method, ε = 0.0001.

(e) Proposed PA-FTS,ε=0.0001, s=10, VA regions=4

PA-FTS [Proposed]

Visual performance results for the 256x256 images with a resolution enhancement factor 4, number LR images = 16, additive Gaussian noise variance = 10,

and Parameters λ = 0.08, β = 1, α = 0.6, P = 2.

Simulation results (cont.)Simulation results (cont.)(a) Original image (b) Bilinearly interp. LR

image with 5 VA regions. (c) FTS SR method (d) SELP-FTS SR method,

ε = 0.0001.(e) Proposed PASR-FTS,

ε=0.0001, s=10, VA regions =4

Visual performance results for the 256x256 images with a resolution enhancement factor 4, number LR images = 16, additive Gaussian noise variance = 10,

and Parameters λ = 0.08, β = 1, α = 0.6, P = 2.

Simulation results (cont.)Simulation results (cont.)

CLOCK image

Number of processed pixels at each iteration


CAMERAMAN image



CARRIER image



PSNR d P t Pi l S i R ltPSNR and Percentage Pixel Savings Results

CARRIER CLOCK PLANE CAMERAMANPSNR(dB)

Pixel Savings

PSNR (dB)

Pixel Savings

PSNR (dB)

Pixel Savings

PSNR (dB)

Pixel Savings

Bicubic 28.6843 0% 24.8088 0% 27.8340 0% 22.4938 0%

FTS 30.8497 0% 27.3246 0% 29.0906 0% 24.6453 0%

SELP-FTS 30.8218 21.67% 27.2717 43.52% 29.0943 36.288% 24.6358 23.9%

PA-FTS 30.6790 69.42% 27.1267 74.4% 28.827 73.92% 24.5765 65.32%

Conducted Subjective TestConducted Subjective TestSet of images used are ‘Carrier’, ‘Clock’, ‘Fighter-plane’ , and ‘Cameraman’ from USC database. p ,From the sequence of the low resolution 64 × 64 images estimate the HR image for a set of four images (magnification factor r = 4) using FTS[Farsiu04] proposed SELP FTS and the proposed[Farsiu04], proposed SELP-FTS, and the proposed PA-FTS method.The SR images are displayed side by side for comparison. Each case is randomly repeated 4 p y ptimes with the left and right images swapped.SR images produced by the perceptual selective methods (SELP or PA-FTS) compared to the non-selective FTS method and rated from 1 5selective FTS method and rated from 1-5 corresponding to ‘worse’, ‘slightly worse’, ‘same’, ‘slightly better’, and ‘better’.

Slightly Worse Same

Slightly Better BetterWorse

(1) (2) (3) (4) (5)

Left ImageQuality

Right ImageQuality

Conducted Subjective TestConducted Subjective Test

The Mean Opinion Score (MOS) is calculated by averaging thecalculated by averaging the responses of all the subjects for each different pair of images.

If MOS > 3 means that selectiveIf MOS > 3 means that selective SELP-FTS and PA-FTS are Better.If MOS < 3 means that nonIf MOS < 3 means that non-selective FTS is Better.

10 Subjects took the test.l

Methods Carrier CLock Fighter Cameraman Average

MOS Results:

SELP-FTS vs. FTS 3.1 2.9 2.95 3.125 3.0187

PA-FTS vs. FTS 3.05 2.925 2.975 3.075 3.0062

Simulation results – MAP based SR

256 × 256 high-resolution images are used to produce, each, a simulated sequence of four 64×64 low resolution images:simulated sequence of four 64×64 low-resolution images:

Blurred with a 4x4 average filter.Shifted by 0.25 pixels in 4 different directions.S b l d b f f 4 i h di iSubsampled by a factor of 4 in each direction.An additive Gaussian noise of variance 4 is added to the resulting LR sequence.g q

Clock-256x256 HR Frame LR Frames Carrier-256x256 HR Frame LR Frames LR FramesParrots-256x256 HR Frame

Simulation resultsMAP SR [Hardie 97]SELP SR [Proposed]PASR [P d]Simulation results

(a) Original image (b) Bilinearly interp. LR image with 3 VA regions.

(c) MAP SR method (d) SELP SR method, ε = 0.0001.

(e) Proposed PASR, ε=0.0001, s=15, VA regions =3

PASR [Proposed]

Visual performance results for the 256x256 CLOCK image with a resolution enhancement factor 4, number LR images = 4, and additive Gaussian noise variance = 4, regularization operator = 150.

Visual performance results for the 256x256 CARRIER image with a resolution enhancement factor 4, number LR images = 4, and additive Gaussian noise variance = 4, regularization operator = 150.

Simulation results (cont.)Simulation results (cont.)(a) Original image (b) Bilinearly interp. LR

image with 4 VA regions. (c) MAP SR method (d) SELP SR method,

ε = 0.0001.(e) Proposed PASR, ε=0.0001,

s=10, VA regions =4

Visual performance results for the 256x256 PARROTS image with a resolution enhancement factor 4, number LR images = 4, and additive Gaussian noise variance = 4, regularization operator = 150.


CLOCK image

(a) SNR gain (b) Number of active pixels at each iteration


CARRIER image

(a) SNR gain (b) Number of active pixels at each iteration


PARROTS image

(a) SNR gain (b) number of active pixels at each iteration


PSNR d P t Pi l S i R lt

CLOCK CARRIER PARROTS

PSNR and Percentage Pixel Savings Results

C OC

PSNR (dB) Pixel Savings PSNR(dB) Pixel

Savings PSNR(dB) Pixel Savings

Bilinear 24.0975 0% 28.4529 0% 22.7470 0%

MAP SR 28.3438 0% 30.5587 0% 24.1490 0%

SELP SR 28.5747 62.32% 30.6353 65.63% 24.2122 49.80%

PASR 28.5324 72.71% 30.5565 74.42% 24.1910 62.76%

Conducted Subjective TestConducted Subjective TestSet of images used are ‘Carrier’, ‘Clock’, ‘Fighter-plane’ from USC database and ‘Parrots’Fighter-plane from USC database and Parrots from LIVE database. From the sequence of the low resolution 64 × 64 images estimate the HR image for a set of fourimages estimate the HR image for a set of four images (magnification factor q = 4) using MAP [Hardie97], the proposed SELP , and the proposed PASR method.p pThe SR images are displayed side by side for comparison. Each case is randomly repeated 4 times with the left and right images swapped.g g ppRight image is compared to the left image and rated from 1-5 corresponding to ‘worse’, ‘slightly worse’, ‘same’, ‘slightly better’, and ‘better’. , , g y ,

Slightly Worse Same

Slightly Better BetterWorse

(1) (2) (3) (4) (5)

Left ImageQuality

Right ImageQuality

Conducted Subjective TestConducted Subjective Test

The Mean Opinion Score (MOS) is calculated by averaging thecalculated by averaging the responses of all the subjects for each different pair of images.

If MOS > 3 means thatIf MOS > 3 means that Proposed PASR is Better.If MOS < 3 means that SELP or MAP SR are BetterMAP SR are Better.

Six Subjects took the test.MOS Results:

Methods Carrier CLock Fighter Parrots Average

MAP vs. PASR 3.792 4.000 4.042 3.708 3.886

SELP vs. PASR 2.92 3.042 2.960 3.042 2.916

References[Itti98] L. Itti, C. Koch, E. Niebur, “A model of saliency-based visual attention for rapid scene analysis,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 20, Issue 11, pp. 1254-1259, Nov. 1998.[Koch06] D. Walther, C. Koch, “Modeling attention to salient proto-objects,” Neural Networks, Vol. 19, n.9, pp. 1395-1407, Nov. 2006.[Ahumada92] A. J. Ahumada, Jr. and H. A. Peterson, "Luminance-model-based DCT quantization for color image compression," SPIE Human Vision, Visual [Farsiu04] S. Farsiu, D. Robinson, M. EladProcessing, and Digital Display III, vol. 1666, pp. 365–374, 1992.[Watson 93] A. B. Watson, "DCT quantization matrices visually optimized for individual images," SPIE Human Vision Visual Processing and Digital Display IV vol 1913 pp 202 216 1993Vision, Visual Processing, and Digital Display IV, vol. 1913, pp. 202–216, 1993., and P. Milanfar, “Fast and robust multiframe super-resolution,” IEEE Transactions on Image Processing, vol. 13, pp. 1327–1344, 2004.[Hardie 97] R. C. Hardie, K. J. Barnard and E. E. Armstrong, “Joint MAP registration and high-resolution image estimation using a sequence of undersampled images ” IEEE Transactions on Image Processing vol 6 no 12 ppestimation using a sequence of undersampled images, IEEE Transactions on Image Processing, vol. 6, no. 12, pp. 1621-33, Dec. 1997.[Ferzli 08] R. Ferzli, Z. A. Ivanovski and L. J. Karam, "An efficient, selective, perceptual-based super-resolution estimator," IEEE International Conference on Image Processing, Oct. 2008. [Sadaka09] N. Sadaka and L. J. Karam, “Efficient, Perceptual, Attentive, Superresolution,” IEEE International[Sadaka09] N. Sadaka and L. J. Karam, Efficient, Perceptual, Attentive, Superresolution, IEEE International Conference on Image Processing, 2009.[Karam 11] L. Karam, N. Sadaka, R. Ferzli and Z. Ivanovski, “An Efficient Selective Perceptual-Based Super-Resolution Estimator,” IEEE Transactions on Image Processing, vol. 20, no. 12, pp. 3470-3482, Dec. 2011.[Sadaka11] N. Sadaka and L. J. Karam, “Efficient Super-Resolution Driven by Saliency Selectivity,” IEEE [S d ] . S d d . J. , c e Supe eso u o ve by S e cy Se ec v y,International Conference on Image Processing, 2011.

super-resolution - lina...

Documents