Bayesian Image Super-resolution, ContinuedLyndsey C. Pickup, David P. Capel, Stephen J. Roberts and Andrew Zisserman, Robotics Research Group, University of Oxford
Goal: Multi-frame super-resolution aims to produce a high-resolution image from a set of low-resolution images by recovering or inventing plausible high-frequency information. With more than one low-resolution image, high-frequency information can be recovered by making use of sub-pixel image displacements. The problem is made more difficult because the sub-pixel displacements -- as well as lighting changes and image blur parameters -- must be recovered from this noisy set of low-resolution input images.
Introduction Key Idea: Integrate over the unknown registrations
Generative model for super-resolution
The goal is to reconstruct the high-resolution image x (right) from a set of noise inputs {y(k)} generated according to the model above.
The image registrations can be viewed as “nuisance parameters” to be integrated out of the super-resolution problem in a Bayesian setting.
Maximum A Posteriori Solution
xMAP using a Huber prior, assuming perfect knowledge of image registrations.
The same problem, but with registrations perturbed by δ ~ N ( 0, 0.12 ) low-resolution pixels.
xMAP using a Gaussian image prior, giving softer edges.
Take a Taylor series expansion of the error in terms δ:
H is a block diagonal sparse matrix, so for many covariance assumptions, S is correspondingly sparse also. Typically C can be assumed diagonal, and L can be solved using gradient descent optimization methods.
Synthetic Data Example
Best Huber-MAP reconstruction. Error compared to known ground truth image.
RMSE = 15.5 grey levels.
Best registration-integrating reconstruction. Note improvement of black-white edge detail in some letters on the bottom line.
RMSE = 14.8 grey levels.
First and last of the 16 input images from the synthetic dataset.
Results on real data Inputs 1 & 10
Registration-Integrating (detail)
Huber-MAP (detail)
Tipping & Bishop(detail), extended to include photometric model.
10 input images, each with 4 registration parameters (two geometric, two photometric).
Huber-MAP: regularizes the smooth areas well, though gives slightly over-crisp edges.
Tipping & Bishop: typical Gaussian over-smoothing, though weakening the prior would risk over-fitting to the noise.
Registration-integrating: shows letters clearly but also gives good regularization on the constant white background areas.
Extra terms as an image prior
The two terms of L involving S can be viewed as an extra prior over the high-resolution image, favouring images which are not acutely sensitive to tiny perturbations in the registrations. Such a perturbation is enough to change the noise pattern on the ML estimate drastically. Two vectors δ1 and δ2 are drawn from N(0,I*0.042) and used to corrupt the registration on a synthetic dataset.
ground truth Input 1/16 Input 16/16 ML1 ML2 abs(ML1-ML2 )
This alternative: accept registration uncertainty, and integrate over the resulting distribution to find super-resolution image directly from inputs.
Most methods, including Tipping & Bishop’s Bayesian Image Super-resolution work, fix the registration first, then compute the high-resolution image.
Assume small perturbations or corrections δ in the set of registration parameters:
Now find –log(p(x|{y(k)})) by integrating over δ:
The regular Huber-MAP objective function for finding x is
where G(x) is the set of image gradients at each pixel, and is the Huber function, defined:
where Images were registered using a simple iterative intensity-based method, typically accurate to within 0.1 pixels. Outputs are at a zoom factor of 4.
Imagex.
Warp with
parameters Blur by
point-spread function.
Decimate by zoom factor.
Corrupt with additive
Gaussian noise.
Lighting changes with
parameters λ
imagey(1)
imagey(K)
There are a number of advantages to integrating over the unknown registrations rather than the high-resolution image in Bayesian image super-resolution:
Integrating over registrations
1a – Can use good image priors such as the Huber prior, which give much better results than Gaussian priors.
1b – A Gaussian prior for the registration offsets is realistic.
2 – The matrix in the log-determinant term of the objective function is small (4*K).
3 – Requires little more memory than the standard Huber-MAP algorithm, so can handle large images.
Integrating over high-resolution image
1 – Must have a Gaussian prior in order for the integral to be tractable. This is not a good model for natural image statistics and tends to blur out the image edges.
2 – The matrix in the log-determinant term of the objective function is large because the matrices tend to be NxN.
3 – The full image-integrating approach can only be run on very small input images (e.g. 9x9 pixels).
Advantages of Registration-Integrating