Image enhancement by non-local reverse heatequation

Antoni Buades1, Bartomeu Coll2, and Jean-Michel Morel3

1 Dpt Matematiques i Informatica Univ. Illes BalearsCtra Valldemossa km 7.5, 07122 Palma de Mallorca, Spain

email: vdmiabc4, tomeu.coll@uib.es2 Centre de Mathematiques et Leurs Applications, ENS Cachan

61, Av du President Wilson, 94235 Cachan, France

Abstract. In 1955 Kovasznay et al. proposed to enhance an image byreversing the heat equation. This process is highly unstable and blowsup the noise. Thus an efficient deblurring depends crucially on accuratedenoising. In this paper we investigate the use of neighborhood filtersand of a recent variant, NL-means, to stabilize the reverse heat equa-tion. We shall prove that adding to the heat equation a term involvingthe self-similarities of the image stabilizes the reverse heat equation. Byexperiments on good quality images, not artificially blurred, we will il-lustrate the feasibility of the method to reveal image details. In contrastwith the various PDE’s for image enhancement, the non-local reverseheat equation thus introduced is linear.

1 Introduction

The difference between the original u0 and a blurred version image of it, k ∗ u0,is roughly proportional to its Laplacian. In order to formalize this far rangingremark, we have to assume that k is spatially concentrated, and we must rescale kas kh(x) = 1

hk( xh

12), with h → 0. Let us denote by x = (x, y) a point of the plane

and assume that u0 is C3 around x. Assume further that k is a positive radialkernel satisfying

∫(1+|x|2+|x|3)k(x)dx < ∞ and

∫x2k(x)dx =

∫y2k(y)dy = 2.

(This last relation can always be imposed, by a rescaling of k.) Under theseassumptions and using a Taylor expansion of u0 around x, it is an easy exerciseto prove that

kh ∗ u0(x)− u0(x)h

→ ∆u0(x)

as h → 0. We can rewrite this relation as

kh ∗ u0(x)− u0(x) = h∆u0(x) + o(h). (1)

Now let u(t,x) denote the solution of the heat equation

∂u

∂t= ∆u, u(0,x) = u0(x).

If u0 is sufficiently smooth, namely C2 and bounded, then we can write

u(t,x)− u(0,x) = t∆u0(x) + o(t). (2)

The comparison of equations (1) and (2) shows that blurring u0 with a kernelkh is for small h equivalent to applying the heat equation to u0 at scale h.These equations have led to an image restoration method: We read in the paper[15] by Lindenbaum, Fischer, and Bruckstein that Kovasznay and Joseph [12]introduced in 1955 the notion that a slightly blurred image could be deblurredby subtracting a small amount of its Laplacian. Numerically, this amounts tosubtracting a fraction λ of the Laplacian of the observed image from itself:

urestored = uobserved − λ∆uobserved.

Dennis Gabor studied this process and determined that the best value of λwas the one that doubled the steepest slope in the image [15]. In other terms onecan to some extent enhance an image by reversing time in the heat equation:

∂u

∂t= −∆u, u(0) = uobserved.

Numerically, this amounts to iterating substraction of its Laplacian from theobserved image. This operation can be repeated several times with some smallvalues of h, until it blows up. Indeed, the reverse heat equation is extremelyill-posed. All the same, this enhancement method is efficient and can be appliedwith some success to most digital images obtained from an optical device. Werefer to [15] for a historical review about this image restoration method.

Since then one can list several attempts to stabilize the time-reverse heatequation or to emulate it by another more stable partial differential equations

The Osher-Rudin “shock filter” The term of shock filter itself was framed byRudin in his PhD dissertation [21], inspired from the use of nonlinear filtersin shock simulation for P.D.E.’s. Osher and Rudin [20] proposed to sharpen ablurred image u0 by applying the following equation:

∂u

∂t= −sign(∆u)|Du|, with u(0,x) = u0(x)

where Du is the spatial gradient of u at x and |Du| its euclidean norm√

u2x + u2

y.This equation can be seen as a pseudo reverse heat equation, where the propa-gation term Du is tuned by the sign of the laplacian.

The Kramer Algorithm. In [13], Kramer defined a filter for sharpening blurredimages. The filter replaces the gray level value at a point x by either the minimumor the maximum of the gray level values in a circular neighborhood B(x, h). Thischoice depends on which is the closest to the current value. This filter is theniterated on the image. Kramer’s filter can be interpreted as a partial differentialequation, by the same kind of heuristic arguments which Gabor developed to

derive the heat equation. It was proved in [23] that the Kramer and the Osher-Rudin filters share the same asymptotic behavior for regular 1D signal. In otherterms, they are infinitesimally identical in 1D. However this is not true in the caseof images. In that case, Guichard and Morel [8] proved that the PDE underlyingthe Kramer filter is

∂u

∂t= −sign(D2u(Du, Du))|Du|.

Thus, the laplacian in the Rudin-Osher equation is replaced by a directionalsecond derivative of the image, D2u(Du, Du). There is a direct link betweenthese filters and two very classical edge detectors, namely the Canny filter andthe zero-crossings of the laplacian. The study in [8] established the followingtable.

In 2D: Kramer Filter Osher-Rudin Filter

∂u∂t

= - |Du|sgn(D2u(Du, Du)) - |Du|sgn(∆u)

replace u by min or max of u min or max of uin a neighborhood in a neighborhood

depending on which of the two on whether the mean of uis the nearest to u in a neighborhood

is above or below u.

Associated edge detector Canny edge detector Zero-Crossing of LaplacianD2u(Du, Du) = 0 ∆u = 0

This table makes clearer the relationships between the various kinds of edgedetectors and deblurring filters. It also specifies in the mid column which kindof local underlying operation is linked with each one of these filters. They can inno way pretend to emulate a reverse heat equation. By the same arguments asOsher-Rudin [20] one sees that Kramer’s operator simply enhances Canny edgesin the same way as the initial shock filter enhances Hildreth-Marr edges.

Before entering the core of the subject, namely how to reverse the heat equa-tion itself, let us point out that there have been other more and more sophisti-cated attempts to perform the inverse heat equation ; see (e.g.o) Hummel et al.[9,10]. And in fact nonlinear local filters for image enhancement were proposedas early as 1951 [28] for television images.

The shock filters asymptotic is also linked to classical reaction-diffusion equa-tions. Alvarez and Mazorra [2] or Price at al. [19] proposed to combine in restora-tion an anisotropic diffusion with a shock filter. Let us also mention [1] for the

combined use of shock filter and diffusion of equations in image quantization :in some extent, quantization acts as a sharpening.

As a further variant on the initial ideas, Perona and Malik in 1987 [17,18]proposed to smooth what needs to be smoothed, namely, the irrelevant homo-geneous regions, and to deblur the boundaries. With this in mind, the diffusionshould look like the heat equation when |Du| is small, but it should act like theinverse heat equation when |Du| is large. Here is an example of a Perona–Malikequation:

∂u

∂t= div(g(|Du|)Du), (3)

where g(s) = 1/(1 + λ2s2). It is easily checked that we have a diffusion equationwhen λ|Du| ≤ 1 and an inverse diffusion equation when λ|Du| > 1. Thus themodel mixes the heat equation and the reverse heat equation.

Variational deblurring (case where the blurring kernel is known). Another classof attempts covers the so called “variational” restoration methods. Their princi-ple is to search for a function urestored which, once blurred by the heat equationor another blurring kernel k, gives the original image uobserved. Since applyingthe heat equation is equivalent to the convolution with an appropriate Gaussianfunction Gσ, one searches for urestored satisfying urestored∗Gσ = uobserved−noise.Because of its instability, this last equation does not yield numerically a uniquesolution. So, among possible solutions urestored is chosen as the most regularones. The regularity is then measured by an energy norm, like the Total Vari-ation [22]. The Rudin–Osher–Fatemi algorithm is efficient when the observedimage u0 is of the form k ∗ u + n, where k and the statistics of the noise n areknown. Given the observed image u0, one tries to find a restored version u thatminimizes the functional

ETV :=∫

Ω

(|Du(x)|+ λ(k ∗ u(x)− u0(x))2)dx, (4)

where Ω denotes the image domain and the parameter λ controls the oscillationin the restored version u. If λ is large, the restored version will closely satisfythe equation k ∗ u = u0, but it may be very oscillatory. If instead λ is small, thesolution is smooth but inaccurate. This parameter can be computed in principleas a Lagrange multiplier.

2 Neighborhood filters and NL-means

Although this may seem paradoxical at first, the deblurring performance isstrongly linked to a denoising performance. This is easily understood in frequencyterms: a deblurring algorithm attempts to divide in the frequency domain theimage by the blurring kernel, which is small on high frequencies. By doing sothe high frequencies of the noise, inherent to image formation, blow up. If wemanage to cure the image of this new added noise, we will be able to go further

into deblurring. However, since most denoising methods also destroy some im-age structures, this cure might pile denoising artifacts on top of the deblurringartifacts. We shall not discuss all denoising methods but concentrate on thosewhich create the least artifacts. Among those, neighborhood filters seem to bethe most adequate as pointed out by several perceptual and structure criteria in[3] and [4]. We shall call neighborhood filters all image filters which reduce thenoise by averaging similar pixels.

2.1 Local neighborhood filters

In order to denoise a pixel, it is better to average the color of this pixel with thenearby pixels with similar colors and only them. This is exactly the technique ofthe sigma-filter. This famous algorithm is generally attributed to J.S. Lee [14]in 1983 but can be traced back to L. Yaroslavsky and the Soviet Union imageprocessing school [27]. The idea is to average neighboring pixels which also havea similar color value. The filtered value by this strategy can be written as

NFh,ρ u(x) =1

C(x)

∫

Bρ(x)

e−|u(y)−u(x)|2

h2 u(y)dy, (5)

where u(x) is the color at x and NFh,ρ u(x) its denoised version. Only pixelsinside Bρ(x) are averaged, h controls the color similarity and C(x) is the nor-malization factor. SUSAN [24] and the bilateral filter [26] make this process moresymmetric by involving a “bilateral” gaussian depending on both space and greylevel. This leads to

SNFh,ρ u(x) =1

C(x)

∫e− |x−y|

2

ρ2 e−|u(y)−u(x)|2

h2 u(y)dy

2.2 Non local averaging

The most similar pixels to a given pixel have no reason to be close to it. Thinkof periodic patterns, or of the elongated edges which appear in most images. In1999 Efros and Leung [6] used non local self-similarities to synthesize texturesand to fill in holes in images. Their algorithm scans a vast portion of the image insearch of all the pixels that resemble the pixel in restoration. The resemblance isevaluated by comparing a whole window around each pixel, not just the color ofthe pixel itself. Applying this idea to neighborhood filters leads to a generalizedneighborhood filter, the non-local means (or NL-means) filter [3][4]. NL-meanshas a formula quite similar to the sigma-filter,

NLu(x) =1

C(x)

∫

Ω

e−(Gρ∗|u(x+.)−u(y+.)|2)(0)

h2 u(y) dy, (6)

where Gρ is the Gauss kernel with standard deviation ρ, C(x) is the normalizingfactor, h acts as a filtering parameter and

(Gρ ∗ |u(x + .)− u(y + .)|2)(0) =∫

R2Gρ(t)|u(x + t)− u(y + t)|2dt.

The formula (6) means that u(x) is replaced by a weighted average of u(y).The weights are significant only if a gaussian window around y looks like thecorresponding gaussian window around x. Thus the non-local means algorithmuses image self-similarity to reduce the noise.

3 Non-local deblurring

3.1 A non-local deblurring energy

Our proposition for deblurring follows from the discussion of the preceding sec-tions. We have seen that all above mentioned PDE’s work as Ersatz for thereverse heat equation but in fact do not hold their promise of performing a re-verse heat equation. As pointed out in Table 1 the various pseudo reverse heatequations or shock filters simply perform a very local and simple operation ona neighborhood. They cannot be qualified of more than enhancement operators.The Osher-Rudin-Fatemi method performs better as pointed out in [8]. Now itssmoothing part, the total variation, is a texture killer. Indeed it removes smallimage oscillations and does not take into account image structure. Thus betteruse an algorithm taking into account fine image structure like NL-means. Nowwe need a variational form for this operator, as pointed out in [11] and [7]. Thetranslation of NL-means in a variational framework is almost straightforward.

The new hypothesis we wish to introduce to justify a nonlocal deblurringmethod is following:

Hypothesis 1 The deblurred image must maintain the same similarities as theblurry image.

It is clear that this hypothesis is a strong limitation since it forbids recreating bydeblurring new structures. On the other hand it permits to rebuild all structuresprovided they left a trace, even tiny, in the blurry image. Thus given a slightlyblurry image u0 we want to define a deblurred version u which has the sameself-similarities as u0. In order to do so, define the new operator associated withu,

NL0u(x) =1

C(x)

∫

Ω

w0(x,y) u(y) dy, (7)

withw0(x,y) = e−

(Gρ∗|u0(x+.)−u0(y+.)|2)(0)

h2 .

This means that NL0u is obtained by applying the NL-means algorithm, butinstead of computing the weights with u they are computed with u0.

We are now in a position to define a non-local deblurring functional.

ENL(u) :=∫ (

(u(x)−NL0u(x))2 + λ(k ∗ u(x)− u0(x))2)dx, (8)

The first term can be viewed as a integral of the Laplacian of u with re-spect to a new metrics defined on the image by the internal similarities of

u0. Thus it forces u to maintain the same coherence as u0 had. For this rea-son it can be viewed as a regularization term. Let us see what link we canmake for this term with more classical regularity term. If for instance we hadw0(x,y) = e−|x−y|2/2h2

(up to a normalization constant) and h is large enough,the first term would boil down to the integral

∫ |∆(x)|2dx. (This point of viewis developed in [25][5].) The derivation of the NL-means from such a variationalprinciple was also commented in [11].

The second term introduces the hypothesized convolution kernel k. In thecase where no particular information is available, k is taken to be a gaussian.The parameter λ is the usual Tikhonov weight parameter between the fidelityto the data term and the regularity term.

3.2 The nonlocal reverse heat equation

One can further simplify the process without loss of generality or precision.Indeed, we have mentioned that λ is left to the user’s choice. Thus it seemsmore sound to view, as the models of section 1 did, the deblurring process asan evolution equation where tiny steps are taken at a time. Here the evolutionequation writes as a non locally stabilized version of the reverse heat equation,namely

∂u

∂t= −∆u + λNL0u, (9)

where λ plays the same role as before and therefore has the same name. Thislinear equation can be by Chernoff’s principle implemented algorithmically byan alternate scheme. One step of reverse heat equation is alternated with onestep of nonlocal regularization. Thus the numerical scheme can be described bythe discrete iteration

un+1 = NL0(un)− c ∗NL0(∆un) = NL0(un − c∆un) (10)

However, the NL-means filter is not fully symmetric and one can have w(x,y) 6=w(y,x) because of the normalization factor C(x) in the definition of w(x,y).Thus the consistency of the above scheme with (9) is not guaranteed unless onesymmetrizes the filter NL. However, the alternate scheme (10) makes sense byitself and this is why it was used in all experiments. The parameter c mustbe small enough to maintain stability CFL conditions. Notice finally that thereverse heat equation and its implementation are linear, in contrast with allabove mentioned PDE’s.

4 Experiments

The non local reverse heat equation performs a uniform enhancement. Pixelswith a similar configuration and therefore likely to belong to the same imagestructure are enhanced uniformly as displayed in Figure 2. In this case, an ar-tificial image with geometric features has been slightly blurred and noisy. The

Fig. 1. Test images used in the experimentation of this paper. Left: Artificial imageslightly convolved with a Gaussian kernel with standard deviation h = 2.5 and withadditive white noise (σ = 2.5). Middle: Lena image. Right: Maillot image. Both naturalimages haven’t been retouched, neither convolved or noisy.

non local reverse heat equation enhances each figure uniformly and controls thenoise at the same time.

In order to control the contrast of the processed image and following Gaborremarks we allow the slope of the edges to be modified only by a factor m =2 or 3. The backward diffusion is stopped at each pixel when the value of theLaplacian ∆u exceeds m ∗∆u0. Figure 3 illustrates how controlling the growthof the Laplacian avoids a blow up of the edges. Due to the uniform enhancementapplied by the NL-means regularization we can increase the contrast by a factorthree without enhancing the noise.

Figure 5 compares the non local reverse heat equation with two classicalequations, the shock filter and the Perona Malik equation. The shock filter en-hances the noise and oscillations of the edges making the result unconvincing.The shock filter can be combined with a mean curvature motion to avoid thiseffect as proposed in [2]. However, the mean curvature motion filters out detailsand texture and rounds off corners as the ones on Figure 2. The Perona-Malikenhances the main edges but filters the rest of the image. This makes this filtersuitable for edge detection but not for enhancement.

5 Conclusion

In this paper we made a quick review of PDE’s emulating a reverse heat equationfor image enhancement, when the blurring kernel is not known. Such algorithmscan be used for enhancing images of relatively good quality by a numerical stepby step procedure equivalent to a PDE. In fact these PDE’s combine a smoothingterm with a reverse equation term but the reverse equation is not a real reverseheat. In continuation we have shown that the smoothing term could be the NL-means algorithm which preserves and enhances all image self-similarities. Thusalternating this filter with a real reverse heat equation permits to control thenoise and yields a enhancement equation. Applying Gabor’s stopping principlewe have shown that the whole process can be stopped on time, thus yielding

Fig. 2. Experiment comparing the reverse heat equation and the modified versionalternating with the NL-means algorithm. Top: Application of the reverse heat equationfor times 0.5, 0.7. Bottom: Modified NL-means reverse heat equation for the same timeand λ = 0.5. The reverse heat equation enhances both the noise and the edges untilthe image blows up. NL-means performs the same enhancement at edges but controlsthe noise in the flat regions. The geometry of the image is preserved by alternatingwith the NL-means even if the contrast at edges quickly increases.

an optimal and artifact free image enhancement. Further work will concentrateon the mathematical analysis of the consistency of a Chernoff iteration. Themethod will also be extended to deal with more specific applications where theblurring kernel and the noise model are known. In that case, as pointed out inthe paper, variational methods have to be involved and not just a step by stepreverse heat equation.

References

1. L. Alvarez and J. Escların, ”Image quantization using reaction-diffusion equations”,SIAM J. Appl. Math., 57 (1997), pp. 153–175.

2. L. Alvarez and L. Mazorra, ”Signal and image restoration using shock filters andanisotropic diffusion”, SIAM J. Numer. Anal., 31 (1994), pp. 590–605.

3. A. Buades, B. Coll and J.M. Morel, ”A review of image denoising methods, with anew one”, Multiscale Modeling and Simulation, vol 4 (2), pp 490-530, 2005.

4. A. Buades, B. Coll and J.M. Morel, ”A non-local algorithm for image denoising”,IEEE Int. Conf. on Computer Vision and Pattern Recognition, 2005.

Fig. 3. Experiment comparing the reverse heat equation and the modified versionalternating with the NL-means algorithm. Top: Application of the reverse heat equationfor times 0.4, 0.5. Middle: Modified NL-means reverse heat equation for the same timeand λ = 0.25. The combination with NL-means performs a better enhancement thanthe reverse heat equation since pixels with a similar structure are uniformly enhanced.However, the contrast at edges ends up growing too much. For this reason, followingGabor’s ideas the reverse heat equation is stopped at each point where the laplacianvalue exceeds m times the original laplacian of the point. Bottom: Application of thereverse heat equation with stopped backward diffusion at m = 2, 3.

Fig. 4. Comparison of the modified reverse heat equation by regularizing with theneighborhood filter and the NL-means algorithm. From left to right: original image, re-verse heat equation with regularization by the neighborhood and the NL-means filters.The NL-means regularization is able to restore the fine texture from the hat while theneighborhood filter blurs it.

5. Coifman, R.R. and Lafon, S. ”Diffusion maps”, Preprint, Jan. 2005.

6. A. Efros and T. Leung, ”Texture synthesis by non parametric sampling,” Proc. Int.Conf. Computer Vision (ICCV 99), vol. 2, pp. 1033-1038.

7. G. Gilboa and S. Osher ”Nonlocal Linear Image Regularization and SupervisedSegmentation”, UCLA CAM Report 06-47, Sept. 2006.

8. Guichard, F. and Morel, J.M., ”A Note on Two Classical Enhancement Filters andTheir Associated PDE’s”, International Journal of Computer Vision, vol. 52, n2,pp. 153–160, 2003.

9. R. A. Hummel, B. Kimia, and S. W. Zucker, ”Gaussian blur and the heat equation:Forward and inverse solutions”, in IEEE Computer Vision and Pattern Recognition,1983, pp. 416–421.

10. R. A. Hummel, B. Kimia, and S. W. Zucker, ”Deblurring gaussian blur”, ComputerVision, Graphics, and Image Processing, 38 (1987), pp. 66–80.

11. Kindermann S, Osher S and Jones PW, ”Deblurring and denoising of images bynonlocal functionals”, Multiscale Modeling and simulation, 4 (4), pp. 1091-1115,2005.

12. L. S. G. Kovasznay and H. M. Joseph, ”Image processing”, Proc. IRE, 43 (1955),p. 560.

13. H. P. Kramer and J. B. Bruckner, ”Iterations of a non-linear transformation forenhancement of digital images”, Pattern Recognition, 7 (1975).

14. J.S. Lee, ”Digital image smoothing and the sigma filter”, Computer Vision, Graph-ics and Image Processing, vol. 24, pp. 255-269, 1983.

15. M. Lindenbaum, M. Fischer, and A. M. Bruckstein, ”On gabor’s contribution toimage enhancement”, Pattern Recognition, 27 (1994), pp. 1–8.

16. M. Mahmoudi and G. Sapiro, ”Fast Image and Video Denoising via Non-LocalMeans of Similar Neighborhoods”, IEEE Signal Processing Letters, vol. 12 (12),2005.

17. P. Perona and J. Malik, ”Scale space and edge detection using anisotropic diffu-sion”, in Proc. Workshop on Computer Vision, Representation and Control, IEEEComputer Society, IEEE Press, Piscataway, NJ, 1987, pp. 16–22.

18. P. Perona and J. Malik, ”Scale space and edge detection using anisotropic diffu-sion”, IEEE Trans. Patt. Anal. Mach. Intell., 12 (1990), pp. 629–639.

Fig. 5. Comparison of different enhancement algorithms. From top to bottom and leftto right: reverse heat equation (t = 0.7), shock filter, the Perona-Malik equation andthe non local reverse heat equation with λ = 0.25 and stopping criterion at threetimes the original laplacian. The shock filter enhances noise and edge irregularities.The Perona-Malik equation enhances the main edges but blurs the rest of the image,thus loosing many details.

19. C.B. Price, P. Wambacq, and A. Oosterlinck. Image enhancement and analysiswith reaction-diffusion paradigm. IEE-P, I:137(3), pp. 136–145, 1990.

20. S. Osher and L. I. Rudin, ”Feature-oriented image enhancement using shock fil-ters”, SIAM J. Numer. Anal., 27 (1990), pp. 919–940.

21. L. I. Rudin, ”Images, Numerical Analysis of Singularities and Shock Filters”, PhDthesis, California Institute of Technology, Pasadena, CA, 1987.

22. L. Rudin, S. Osher and E. Fatemi, ”Nonlinear total variation based noise removalalgorithms”, Physica D, 60, pp. 259-268, 1992.

23. Schavemaker, J.G.M., Reinders, M.J.T., Gerbrands, J.J. and Backer, E. ”Imagesharpening by morphological filtering”, Pattern Recognition, vol. 33, n6, pp. 997–1012, 2000.

24. S.M. Smith and J.M. Brady, ”Susan - a new approach to low level image process-ing”, International Journal of Computer Vision, Volume 23 (1), pp. 45-78, 1997.

25. Szlam, A.D., Maggioni, M., Coifman, R.R. and Bremer Jr, J.C. ”Diffusion-drivenmultiscale analysis on manifolds and graphs: top-down and bottom-up construc-tions, Sensor bias estimation from measurements of targets with known determinis-tic dynamics”. Edited by Burns, Paul; Blair, William D. Proceedings of the SPIE,5914, pp. 438–448, 2005.

26. C. Tomasi and R. Manduchi, ”Bilateral filtering for gray and color images,” SixthInternational Conference on Computer Vision, pp. 839-46. 1998.

27. L.P. Yaroslavsky, ”Digital Picture Processing - An Introduction”, Springer Verlag,1985.

28. P.C. Goldmark and J.M. Hollywood ”A new technique for improving the sharpnessof television pictures”, pp. 1314–1322, Proc. IRE, Oct. 1951.