maximum likelihood estimation of regularisation ⊠icip vidal 3oct18.pdf · imaging inverse...
TRANSCRIPT
![Page 1: MAXIMUM LIKELIHOOD ESTIMATION OF REGULARISATION ⊠ICIP Vidal 3oct18.pdf · Imaging Inverse Problems ... The challenge is that ð( | ) is intractable as it involves solving two](https://reader035.vdocuments.mx/reader035/viewer/2022070909/5f90414f41d7676ea10663dd/html5/thumbnails/1.jpg)
Imaging Inverse Problems
OBJECTIVE: to estimate an unknown image ð¥ from an observation ðŠ.
A +
ð§~ð(0, ð2ðŒ)
ð¥ â âð
UNKNOWNIMAGE
ðŠ â âð
OBSERVATIONLINEAR
OPERATOR
NOISE
Some canonical examples:⢠image deconvolution ⢠compressive sensing ⢠super-resolution ⢠tomographic reconstruction⢠inpainting
CHALLENGE: not enough information in ðŠ to accurately estimate ð¥.
For example, in many imaging problems ðŠ = ðŽð¥ + ð§ where the operator ðŽ is either:
High-dimensional problems â ð~106
RANK-DEFICIENT
SMALL EIGENVALUESðŽ
not enough equations to solve for ð¥
NOISE AMPLIFICATION unreliable estimates of ð¥
REGULARISATION: We can render the problem well-posed by using prior knowledge about theunknown signal ð¥.
How do we choose the regularisation parameter ð?The regularisation parameter controls how much importance we give to prior knowledge andto the observations, depending on how ill posed the problem is and on the intensity of the noise.
ðœ
PRIORKNOWLEDGE
OBSERVEDDATA
POSSIBLE APPROACHES FOR CHOOSING ð:
NON BAYESIAN⢠Cross-validation â Exhaustive search method
⢠Discrepancy Principle
⢠Stein-based methods âMinimise Steinâs Unbiased Risk Estimator (SURE), a surrogate of the MSE
⢠SUGAR â More efficient algorithms that uses gradient of SURE
Limitation: mostly designed for denoising problems. Difficult to implement
BAYESIAN⢠Hierarchical â Propose prior for ð and work with hierarchical model
⢠Marginalisation â Remove ð from the model ð(ð¥|ðŠ) = +â ð ð¥, ð ðŠ ðð
Limitation: only for homogeneous ð(ð¥) or cases with known ð¶(ð)
⢠Empirical Bayes â Choose ð by maximising marginal likelihood ð(ðŠ|ð)
Difficulty: ð(ðŠ|ð) becomes intractable in high-dimensional problems
Our strategy: Empirical Bayes
The challenge is that ð(ðŠ|ð) is intractable as it involves solving two integrals in âð (to marginalise ð¥ and to compute
ð¶ ð ). This makes the computation of ð ðð¿ðž extremely difficult.
OUR CONTRIBUTIONWe propose a stochastic optimisation scheme to compute the maximum marginal likelihood estimator of theregularisation parameter. Novelty: the optimisation is driven by proximal Markov chain Monte Carlo (MCMC) samplers.
We want to find ð that maximises the marginal likelihood ð(ðŠ|ð):
àµ
ð(ðŠ|ð) = âð ð ðŠ, ð¥ ð ðð¥
ððð¿ðž = ðððððð¥ð â Î
ð(ðŠ|ð)
We should be able to reliably estimate ð from ðŠ as ðŠ is very high-dimensional and ð is a
scalar parameter
Proposed stochastic optimisation algorithmIf ð(ðŠ|ð) was tractable, we could use a standard projected gradient algorithm:
STOCHASTIC APPROXIMATION PROXIMAL GRADIENT (SAPG) ALGORITHM
ðð¡+1 = ððððŠÎ
[ðð¡ + ð¿ð¡ð
ððlog ð(ðŠ|ðð¡)] where ð¿ð¡ verifies
To tackle the intractability, we propose a stochastic variant of this algorithm based on a noisy estimate of ð
ððððð(ð(ðŠ|ðð¡). Using Fisherâs identity we have:
ð
ððlog ð ð ð = ðžð|ð,ð
ð
ððlog ð ð, ð ð = ðžð|ð,ð âð ð¥ â
ð
ððð¿ðð ð¶ ð
If ð¶ ð is unknown we can use the identity â ð
ððð¿ðð ð¶ ð = ðžð|ð ð ð
The intractable gradient becomes ð
ððlog ð ð ð =ðžð|ð,ð âð ð¥ + ðžð|ð ð ð
Now we can approximate ðžð|ð,ð âð ð¥ and ðžð|ð ð ð with Monte Carlo estimates.
We construct a Stochastic Approx. Proximal Gradient (SAPG) algorithm driven by two Markov kernels Mð and Kðtargeting the posterior ð ð, ð ð and the prior ð ð¥ ð respectively.
ð¿ð+ð~Mðð¡ X ðŠ, ðð¡ , ðð¡
ðð¡+1 = ððððÎ
ðð¡ + ð¿ð¡ ð ðŒð+ð â ð ð¿ð+ð
ðŒð+ð~Kðð¡ U ðð¡ , ðð¡
ð ð¥ ðŠ, ððð¿ðž
ð ð¥ ððð¿ðž
ððð¿ðž
CONVERGEJOINTLY TO
EMPIRICAL BAYES POSTERIOR
STOCHASTIC GRADIENT
How do we generate the samples?
MOREAU-YOSIDA UNADJUSTED LANGEVIN ALGORITHM (MYULA)
ð¥ð¡+1 = ð¥ð¡ â ðŸ (ð»ððŠ ð¥ð¡ +ð¥ð¡ â ðððð¥ð
ðð ð¥ð¡
ð) + 2ðŸ ðð¡+1
ð» ððð(ð¥)
We use the MYULA algorithm for the Markov kernels Kð and Mð because they can handle:⪠High-dimensionality !⪠Convex problems with a non-smooth ð(ð)
⪠The MYULA kernels do not target ð ð¥ ðŠ, ð and ð ð¥ ð exactly.⪠Sources of asymptotic bias:
⪠Discretisation of Langevin diffusion: controlled by ðŸ and ð.⪠Smoothing of non differentiable ð(ð¥): controlled by ð.
⪠ðŸ,ð must be < inverse of Lipchitz constant of the gradient driving each diffusion respectively.⪠More information about how to select each parameter can be found in [2].
WHERE DOES MYULA COME FROM?
ðð ð¡ =1
2ð» log ð ð ð¡ |ðŠ ðð¡ + ðð ð¡
LANGEVIN DIFFUSION
ð¥ð¡+1 = ð¥ð¡ + ðŸ ð» ð¿ðð ð ð¥ð¡ ðŠ, ð + 2ðŸ ðð¡+1
ðð¡+1~ð 0, ð
UNADJUSTED LANGEVIN ALGORITHM (ULA)
If ð(ð¥) is not Lipschitz differentiable ULA is unstable.
The MYULA algorithm uses Moreau-Yosida regularization to replace the non-smooth term ð(ð¥) with its Moreau Envelope ðð(ð¥).
Moreau-Yosida regularisation controlled by ð
ð ð¥ â ðâ|ð¥|
ðð ð¡
ðð¡â
ð¥ð¡+1âð¥ð¡
ðŸ
EULER MARUYAMA APPROXIMATION
â exp{ â ððŠ(ð¥) â ð ð(ð¥) } BROWNIAN MOTION
ResultsWe illustrate the proposed methodology with an image deblurring problem using a total-variation prior.
We compare our results with those obtained with:⪠SUGAR⪠Marginal maximum-a-posteriori estimation⪠Optimal or oracle value ðâ
EVOLUTION OF ð THROUGHOUT ITERATIONS
BOAT IMAGE
QUANTITATIVE COMPARISON
â Generally outperforms the other approaches
For each image, noise level, and method, we compute the MAP estimator and we display on this table the average results for the 6 test images.
EB performs remarkably well at low SNR values
Longer computing time only justified if marginalisation canât be used
SUGAR performs poorly when the main problem is not noise
Close-up for SNR=20db
METHOD COMPARISON
Over-smoothing: too much regularisation
Ringing:
not enough regularisation
PRIOR
DATA
DEGRADED EMPIRICAL BAYES MAP ESTIMATOR
SNR=40db
Future work⪠Detailed analysis of the convergence properties.
⪠Extending the methodology to problems involvingmultiple unknown regularisation parameters.
⪠Reducing computing times by accelerating the Markovkernels driving the stochastic approximation algorithm:
⪠Via parallel computing⪠By choosing a faster Markov kernel
⪠Adapt algorithm to support some non-convex problems.
⪠Use the samples from ð ð ð, ðœ ðŽð³ð¬ to perform
uncertainty quantification.
References[1] Marcelo Pereyra, José M Bioucas-Dias, and Mário ATFigueiredo, âMaximum-a-posteriori estimation with unknownregularisation parameters,â in 23rd European Signal ProcessingConference (EUSIPCO), 2015. IEEE, pp. 230â234, 2015.
[2] Alain Durmus, Eric Moulines, and Marcelo Pereyra, âEfficientBayesian computation by proximal Markov Chain Monte Carlo:when Langevin meets Moreau,â SIAM Journal on ImagingSciences, vol. 11, no. 1, pp. 473â506, 2018.
[3] Ana Fernandez Vidal and Marcelo Pereyra. "MaximumLikelihood Estimation of Regularisation Parameters." In 2018 25thIEEE International Conference on Image Processing (ICIP), pp.1742-1746. IEEE, 2018.
MAXIMUM LIKELIHOOD ESTIMATION OF REGULARISATION PARAMETERSAna Fernandez Vidal, Dr. Marcelo Pereyra ([email protected], [email protected])
School of Mathematical and Computer Sciences, Heriot-Watt University, Edinburgh EH14 4AS United Kingdom
The observation ð is related to ð by a statistical model as a random variable.
We use a prior density to reduce uncertainty about and deliver accurate estimates.
The Bayesian Framework
ð(ð¥) penalises undesired properties and the regularisation parameter ð controls the intensity of the penalisation.
Example: random vector ðrepresenting astronomical images.
ð ð¥|ð =ðâð ð ð¥
ð¶(ð)
HYPERPARAMETER
We model the unknown image ð¥ as a random vector with
prior distribution ð ð¥|ð promoting desired properties about ð¥.
PRIOR INFORMATION
ð ðŠ ð¥ â ðâððŠ(ð¥)
The observation ðŠ is related to ð¥ by a statistical model:
OBSERVED DATA
LIKELIHOOD
Observed and prior information are combined by using Bayes' theorem:
POSTERIOR DISTRIBUTION
ð ð¥ ðŠ, ð â ðâ ððŠ ð¥ + ð ð ð¥
ð ð¥ ðŠ, ð = ð ðŠ ð¥ ð ð¥|ð /ð(ðŠ|ð)
à·ð¥ððŽð = argmaxð¥ââð
ð ð¥ ðŠ, ð = argminð¥ââð
ððŠ ð¥ + ð ð(ð¥)
The predominant Bayesian approach in imaging is MAP estimationwhich can be computed very efficiently by convex optimisation:
We will focus on convex problems where:- ð(ð¥) is lower semicontinuous,
proper and possibly non-smooth- ððŠ(ð¥) is Lipschitz differentiable
with Lipschitz constant L
ððŠ(ð¥) = ðŠâðŽð¥ 2
2
2ð2)ðŠ~ð(ðŽð¥, ð2ðŒ
NORMALIZING CONSTANT
Conclusions⪠We presented an empirical Bayesian method to estimate regularisation parameters in convex inverse imaging problems.⪠We approach an intractable maximum marginal likelihood estimation problem by proposing a stochastic.
optimisation algorithm.⪠The stochastic approximation is driven by two proximal MCMC kernels which can handle non-smooth regularisers efficiently.⪠Our algorithm was illustrated with non-blind image deconvolution with TV prior where it:
â achieved close-to-optimal performanceâ outperformed other state of the art approaches in terms of MSEÃ had longer computing times.
⪠More details can be found in [3].
EXPERIMENT DETAILSâ¢Recover ð from a blurred noisy observation ð , ð = ðŽð + ð and ð )~N(0, ð2ðŒâ¢ðŽ is a circulant uniform blurring matrix of size 9x9 pixelsâ¢ð ð = ðð ð â isotropic total-variation pseudo-normâ¢We use 6 test images of size 512x512 pixelsâ¢We compute the MAP estimator for each using different values of ð obtained with different
methodsâ¢We compare methods by taking average value over 6 images of the mean squared error
(MSE) and the computing time (in minutes)
ORACLE ORIGINAL
in minutes
à Increased computing timesâ Achieves close-to-optimal performance
SNR=40dB
Close-up for SNR=20dB
DEGRADEDORIGINAL EMP. BAYES MARGINAL MAP SUGAR