wavelet domain image restoration with adaptive edge-preserving regularization

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 9, NO. 4, APRIL 2000 597

Wavelet Domain Image Restoration with AdaptiveEdge-Preserving Regularization

Murat Belge, Misha E. Kilmer, and Eric L. Miller, Member, IEEE

Abstract—In this paper, we consider a wavelet based edge-pre-serving regularization scheme for use in linear image restorationproblems. Our efforts build on a collection of mathematical resultsindicating that wavelets are especially useful for representingfunctions that contain discontinuities (i.e., edges in two dimensionsor jumps in one dimension). We interpret the resulting theoryin a statistical signal processing framework and obtain a highlyflexible framework for adapting the degree of regularization tothe local structure of the underlying image. In particular, we areable to adapt quite easily to scale-varying and orientation-varyingfeatures in the image while simultaneously retaining the edgepreservation properties of the regularizer. We demonstrate ahalf-quadratic algorithm for obtaining the restorations fromobserved data

Index Terms—Image restoration, regularization.

I. INTRODUCTION

I N MANY applications, recorded images represent a de-graded version of the original scene. For example, the

images of extraterrestrial objects observed by ground basedtelescopes are distorted by atmospheric turbulence [1] whilemotion of a camera can result in an undesired blur in a recordedimage. Despite the different origins, these two cases along withothers from a variety of fields, share a common structure wherethe exact image undergoes a “forward transformation” and iscorrupted by observation noise. The source of this noise is thedisturbance caused by the random fluctuations in the imagingsystem and the environment. The goal of image restoration is torecover the original image from these degraded measurements.

Often, the forward transformation acts as a smoothing agentso that the resulting restoration problem is ill-posed in the sensethat small perturbations in the data can result in large, non-physical artifacts in the recovered image [2]. Such instabilityis typically addressed through the use of a regularization pro-cedure which introducesa priori information about the orig-inal image into the restoration process. The prior information

Manuscript received August 18, 1998; revised August 9, 1999. This workwas supported by an ODDR&E MURI under Air Force Office of ScientificResearch contract F49620-96-1-0028, a CAREER Award from the NationalScience Foundation MIP-9623721, and the Army Research Office DeminingMURI under Grant DAAG55-97-1-0013. The associate editor coordinating thereview of this manuscript and approving it for publication was Prof. Patrick L.Combettes.

M. Belge is with Aware, Inc., Bedford, MA 01730 USA.M. Kilmer is with the Department of Mathematics, Tufts University, Medford,

MA 02155 USA.E. L. Miller is with the Electrical and Computer Engineering De-

partment, Northeastern University, Boston, MA 02115 USA (e-mail:[email protected]).

Publisher Item Identifier S 1057-7149(00)02674-9.

underlying the most commonly used regularization schemes isthat the image is basically smooth [2]. While the regularizedrestorations are less sensitive to noise it is well known that thesmoothness assumption impedes the accurate recovery of im-portant features, especially edges.

In response to this problem, there has recently beenconsiderable work in the formulation of “edge-preserving”regularization methods which result in less smoothing toareas with large intensity changes in the restored image.These methods necessarily require nonquadratic regularizationfunctions and therefore result in nonlinear image restorationalgorithms. Along these lines, Geman and Yang [3] introducedthe concept of “half quadratic regularization” which addressesthe nonlinear optimization problem that results from using suchfunctions. Later, Charbonnieret al. [4] built upon the resultsof this work by providing the conditions for edge preservingregularization functions. Another recent advance in this area isthe total variation (TV) based image restoration algorithms [5].In this approach, images are modeled as functions of boundedvariation which need not be continuous. Therefore, formationsof edges are encouraged and the restorations obtained bythe TV based algorithms look sharper than those obtainedby conventional techniques, especially if the exact image ispiecewise continuous.

In this work, we consider a statistically based, wavelet-do-main approach to edge-enhanced image restoration in which weemploy a stochastic interpretation of the regularization process[6]. We note that most of the work to date on wavelet-based,statistical regularization methods has concentrated on the useof multiscale smoothness priors [9]–[12]. While Wanget al.didconsider issues of edge preservation in [12], their method wasbased on the processing of the output of an edge detector appliedto the noisy data to alter the degree of regularization in a mul-tiscale smoothness constraint. As described below and in sub-sequent sections, our approach is significantly different as theedge preservation is built directly into the regularization schemeitself.

Specifically, we regard the image as a realization of arandom field for which the wavelet coefficients are inde-pendently distributed according to generalized Gaussian (GG)distribution laws. This model is motivated by two factors.First, recent work [13], suggests that these models, whichhave heavier tails than a straight Gaussian distribution, pro-vide accurate descriptions of the statistical distribution ofwavelet coefficients in image data. Second, in addition tobeing a basis for , wavelets also are unconditionalbases for more exotic function spaces whose members in-clude functions with sharp discontinuities and thus serve as

1057-7149/00$10.00 © 2000 IEEE

598 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 9, NO. 4, APRIL 2000

natural function spaces in which to analyze images [15].Because the norms in these Besov spaces are nothing morethan weighted , , norms of the wavelet coefficients,it is shown easily that deterministic regularization with aBesov norm constraint is equivalent to the specification of anappropriately parameterized GG wavelet prior model. Fromthis perspective, our work can be viewed as an extensionof the research done mostly in the area of image denoising.Specifically, the wavelet domain image model of interest inthis paper and the resulting nonlinear restoration algorithmare related to the large body of work originating from thewavelet shrinkage estimatorsfirst proposed by Donoho andJohnstone [18]. In a series of papers, Donoho and Johnstonehave shown that wavelet shrinkage estimators achieve nearoptimal estimation performance when the unknown signalbelongs to Besov spaces. Later, several authors contributedto the advancements in the area. The notion of Besov regu-larization has been introduced by Amato and Vuza [17] andChambolle, DeVore, Lee, and Lucier [8] and the resultingtheory was interpreted in a function space setting. On theother hand, Simoncelli and Adelson [6] developed a sim-ilar denoising scheme, which they calledBayesian waveletcoring, by stochastically modeling the image subbands.

In this work, we make use of GG wavelet priors in a numberof ways. We show that their use in an image restoration problemdoes in fact significantly improve the quality of edge informa-tion relative to more common smoothness priors. Inspired by the“lagged diffusivity” fixed point iteration proposed by Vogel andOman [19] for the solution of the TV problem, we also providean efficient algorithm for solving the nonlinear optimizationproblem defining the restoration. By appropriately structuringthe weighting pattern on the wavelet norm, we demonstratethat these models provide an easy and flexible framework foradaptively determining the appropriate level of regularizationas a function of the underlying structure in the image; in partic-ular, scale-to-scale or orientation based features. This adaptationis achieved through a data-driven choice of a vector of regular-ization parameters. For this task, we introduce and make useof a multivariate generalization of the-curve method devel-oped in [20] for choosing a single regularization parameter. Weverify the performance of this restoration scheme on a variety ofimages, comparing the results both to smoothness constrainedmethods and the TV restorations.

We recognize that there are asymptotic results which state thatthe -curve does not provide consistent estimates of the regu-larization parameters either as the noise level goes to zero orthe data length goes to infinity [21], [22]. In the nonasymptoticregime however, empirical results do point to the practical utilityof this method. Moreover, as described in [20], the-curveframework is easily adapted to handle multiple regularizationparameters, a feature required for the work here.

The remainder of this paper is organized as follows. In Sec-tion II we give the wavelet domain formulation of the imagerestoration problem. In Section III we introduce a multiscaleprior model for images and use this model in Section IV to de-velop an image restoration algorithm. In Section V we applythe “ -hypersurface” method to the simultaneous multiple pa-rameter selection problem posed by our image restoration al-

gorithm. In Section VI we demonstrate the effectiveness of ouralgorithm by comparing our results with existing image restora-tion schemes. Finally, in Section VII, conclusions and futurework are discussed.

II. REGULARIZED IMAGE RESTORATION

A grey-scale image, , can be considered as a collection ofpixels obtained by digitizing a continuous scene. The image isindexed by , , and the intensity at theposition is denoted by . In image reconstruc-tion and restoration problems, the objective is to estimate theimage from its degraded measurements. Mathemati-cally, such a scenario can be adequately represented by the fol-lowing linear formulation:

(1)

where the vectors, and represent, respectively, the lexico-graphically ordered degraded image, the original image, and thedisturbance. The known square matrixrepresents the lineardistortion. is typically ill-conditioned. This implies that theexact solution, , to (1) is extremely contaminatedby noise. Rather, a unique and stable estimateis sought byincorporating prior information on the original image. This hasthe effect of replacing the original ill-conditioned problem witha well-conditioned one whose solution approximates that ofthe original. Such a technique is called a regularization method[23]–[26].

In the Bayesian image restoration method of interesthere, the prior information is quantified by specifying aprobability density on and combining this with the in-formation contained in to produce an estimate of theunknown image. We assume here a linear, additive whiteGaussian noise model so that the probability density foris

where is the number of pixels in the image andis the noise variance. If it so happens that the probability

distribution for is in the formthen by Bayes’s rule, the MAP estimate,, is obtained byminimizing the following log-posterior density with respect to

[27], [28]

(2)

The function , called theenergyfunction in the contextof Bayesian estimation, is the energy attributed to the image,and is the vector of possibly unknown model parameters. Wegive low energy to the images which coincide with our priorconceptions and high energy to those which do not. Thus, if ourprior belief about the image is that the original image is smooth,then the energy is a measure of theroughness.

A. Wavelet Representation of Image Restoration Problem

In this paper, we adopt a wavelet domain approach to theimage restoration problem. A comprehensive introduction to thewavelet theory can be found in [14], [15], and [29]. It is possibleto obtain the wavelet transform of images through a separablerepresentation. This decomposition can be implemented by one-

BELGE et al.: WAVELET DOMAIN IMAGE RESTORATION 599

Fig. 1. One-level wavelet decomposition of an image.

dimensional (1-D) filtering of rows and columns of images. InFig. 1, we have schematically illustrated a one-level wavelet de-composition of an image with denoting thefinest scale scaling coefficients. The one-level wavelet decom-position of the image produces four subimages ofsize , . represents the

scaling coefficients at scale and arethe wavelet coefficients at scale corresponding to the ver-tical, horizontal and diagonal orientations in the image plane.Multilevel wavelet decompositions of the image canbe obtained by applying the one-level wavelet decompositionscheme, outlined above, recursively to the scaling coefficients

. For an -level wavelet decomposition,denotes the lowest resolution at which the image is represented.We will use to denote the vector of wavelet (scaling) co-efficients obtained by lexicographically ordering the elementsof the two-dimensional (2-D) array and to denotea lexicographically ordered version of all wavelet coefficients

.With the conventions above, we can represent the problem in

(1) in the wavelet domain as

(3)

where is the 2-D wavelet transform matrix,, and arethe vectors holding the scaling and wavelet coefficients of thedata, the original image, and the disturbance,is the waveletdomain representation of our linear degradation operator, and

follows from the orthogonality of the wavelet trans-form. Note that since the wavelet transform is orthonormalisagain Gaussian with zero mean and variance.

III. M ULTISCALE IMAGE MODEL

A key component of our image restoration algorithm is theuse of a multiscale stochastic prior model for. To motivatethe particular choice of prior model used here, consider thewavelet coefficients of a typical image at a particular resolution.Wavelet coefficients are obtained by differentiation-like opera-tions. Since the spatial structure of many images typically con-sists of smooth areas dispersed with occasional edges, the distri-bution of wavelet coefficients should be sharply peaked aroundzero, due to the contribution of smooth areas, and have broadtails representing the contribution of the edges [6].

Following the work in [6] and [7] on image coding and de-noising, we model the distribution of wavelet coefficients of im-ages by a generalized Gaussian () density [13], [14]

(4)

where is a parameter which determines the tail be-havior of the density function and is ascale parametersim-ilar to the standard deviation of a Gaussian density. We will referto the zero mean density in (4) as . For wehave the Laplacian density and for we have the familiarGaussian density. The tails of the GG distribution becomes in-creasingly heavy asapproaches zero. We assume that the meanof the image is subtracted from the image and that the scalingcoefficients , are i.i.d. .

The specification of one parameter for each scale and orien-tation results in an image model far too complex to be of use in arestoration procedure. Nonetheless, the structure of the model in(4) coupled with the specification of the problem in the waveletdomain does suggest a variety of simplifications which are ofuse for the restoration problem. In this work, we consider thefollowing three models.

1) Model 1: Scaling coefficients , are i.i.d.

with and the wavelet coefficientsare i.i.d. with exponentially decreasing variances,i.e.

with the coarsest scale,the scale parameter corresponding toand .The rationale behind this model is that it is equivalentto a deterministic modeling of the image as a memberof a Besov space [15].

2) Model 2: Scaling coefficients , are

i.i.d. with and the wavelet co-efficients at a particular scale are i.i.d. with

. This model isuseful in cases where the variance of the waveletcoefficients at different scales cannot be well-approxi-mated by a simple exponential law.

3) Model 3: Scaling coefficients , are i.i.d.

with and the wavelet coefficients atdifferent orientations (horizontal, vertical or diagonal)are distributed with

. Such a model is most suitablefor images with significantly different characteristics


in different orientations as might arise in geophysicalrestoration problems involving layered structures.

We make several observations regarding these models. First,they are indeed of low dimensionality. In addition to theand

parameters, Model 1 is characterized by twocoefficients:one for the coarsest scale scaling coefficients and one multi-plying the exponential for the wavelet coefficients. There are atotal of ’s for Model 2 and four values requiredto characterize Model 3. In subsequent sections, we shall seethat the number of regularization parameters to be determinedin the restoration algorithm is equal to the number of’s char-acterizing the prior model being used. Moreover, an appropriateon-line choice of the model parameters provides a mechanismfor adapting the level of regularization in an image to the un-derlying scale-to-scale structure (Models 1 and 2) or to orien-tation-dependent structure (Model 3). While, the above threemodels certainly do not represent an exhaustive enumeration ofall possible multiscale regularization approaches, as seen in Sec-tion VI, they do provide a strong indication as to the utility ofthis type of modeling technique for image restoration.

Finally, we comment on the estimation of the hyperparam-eters, , , and . In a restoration algorithm, these param-eters could be estimated from the data by assigning appro-priate priors to each and maximizing the resulting log-pos-terior function with respect to the hyper-parameters and theimage. However, such an approach presents many computa-tional difficulties and unnecessarily complicates the problem.Instead, for the remainder of this paper we choose to sim-plify the problem by fixing the and a priori. Generally,the performance of the regularizer is impacted to a greaterextent by the on-line identification of the parameters [30](or as explained in subsequent sections, quantities closely re-lated to ) so we concentrate our effort on identifying goodchoices of . The issue of selecting an appropriateis ex-tensively discussed in Section IV-A. As for the selection of

, we propose using a fixeda priori choice obtained fromthe empirical study of a number of images. According to ourfindings, for most images representing natural scenes thevalue which produced the best fit to the image data underthe Model 1 scheme (for ) fell between 0.6–1.6 withmean . We evaluated the effects of varying thevalue on a number of restoration problems and saw that theresults were relatively insensitive to variations in in therange suggested by the observations. The first example inSection VI supports this. Note also that past research revealssimilar conclusions [30] indicating that the performance ofthe estimator is degraded little by the error in. Therefore,for all experiments performed we used as our fixeda priori choice.

IV. M ULTISCALE IMAGE RESTORATIONALGOITHM

The MAP estimate of the wavelet coefficients of the orig-inal image is found by maximizing the log-posterior functionin (2). Substituting the prior probability density developed inSection III into (2), the MAP estimate of is seen to be the

minimum of the following cost function with respect to(as-suming for the time being that is known)

(5)where are weighting parameters and

. The formulation in (5) easily ac-commodates the Models 1–3 regularization schemes discussedin Section III by defining the appropriate relationships for .

For example, putting and results inthe Model 1 regularization scheme while assigning a different

to each scale in the wavelet domain without regarding theorientation we obtain the Model 2 regularization scheme. Sup-pose that has a minimum in , then at a stationary point

, the gradient of must vanish. Unfortunately, thenorm terms appearing in (5) is not differentiable for .Hence, we propose the followingsmoothapproximation to the

norm, raised to the power, as in [19]

(6)

where is a stabilization constant and denotes thethelement of the vector. Substituting (6) into (5) and taking thegradient of the cost function we arrive at the following equation

(7)

(8)

where is the minimum of with the approximationin (6), is the th element of , and is the associated reg-ularization parameter. The above equation gives the first orderconditions that must be satisfied by. By direct analogy withthe lagged diffusivity method of Vogel and Oman [19], we candevelop a fixed point iteration to solve for . Starting with an

initial point , we solve the following equation for :

(9)

where is obtained by replacing by in (9). The iteration

is terminated whenever , with beinga small positive constant. We use in our simulations.The fixed point iteration in (9) is a special case of the “halfquadratic regularization” scheme introduced by Gemanet al.[3]and the ARTUR scheme due to Charbonnieret al.[4]. Adoptingthe notation in [4] we define the following function:

(10)

Then, the approximated cost function can be expressed in termsof the function . Furthermore, satisfies the conditionsa)–i) presented in [4, Eq. (12)]. Roughly speaking, these condi-tions ensure that the large-magnitude wavelet coefficients, pri-marily associated with edges, are penalized less than the small-


magnitude wavelet coefficients and that the restoration algo-

rithm is convergent in the sense that the sequence is

convergent and that . In the special case

where is convex (which occurs if ) and is full-rank,

the iterates converge and the computed solution is the uniqueminimum of (5). However, when , is nonconvex thealgorithm computes a local minimum of (5) [4].

The iterative algorithm in (9) requires the solution of a verylarge linear matrix equation. Note that the matrix appearing onthe right hand side of (9) is symmetric and positive definite.Therefore, the conjugate gradient (CG) algorithm [31] can be

conveniently used to compute the solution in (9) at eachstep. In this way, the algorithm given in (9) is doubly iterative in

that an outer iteration is used to update the solutionand aninner iteration is used to solve the system of equations in (9) bythe CG method. The special structure of the matricesandcould be used to decrease the computational cost substantially.The first matrix, is merely the wavelet domain representationof our degradation operator. If the kernel is convolutional, ithas been shown by Zervakiset al. [32] that this matrix can bediagonalized by a special Fourier transform matrix by invokingthe circulant assumption. On the other hand, the second matrix

is diagonal in the wavelet domain. Therefore, the vectormatrix multiplications required for the implementation of theCG algorithm can be computed in an efficient way by goingback and forth between the wavelet and the Fourier transformdomains. In this case, the cost of multiplying a vector with the

matrix is dominated by the cost of the FFTwhich is .

We note that the iterative algorithm in (9) can be efficienteven in the case where is not convolutional since the waveletdomain representation of a wide range of operators is sparse[18]. In those cases, standard techniques for sparse matrices canbe used to reduce computational complexity.

A. Selection of Appropriate and values

The possibility that multiple local minima of (5) may existfor presents an interesting tradeoff. From a computa-tional viewpoint, it is highly desirable to use , since inthis case the cost function is convex and global convergence isguaranteed. However, based on empirical studies of the waveletcoefficients of images it has been shown that the GG model forthe distribution of the wavelet coefficients usually correspondsto and a typically recognized value is [13].Analysis of the use of GG priors in the context of image de-noising has been performed by Moulin and Liu [33]. The resultsin [33] suggest that only modest improvement can be achievedby using as compared with . In our experiments,we essentially arrived at the same conclusion. That is, the re-stored images obtained by using the best value of(in terms ofmodel fit) were visually almost the same as the results obtainedby using , although slightly lower estimation errors wereobserved for . Therefore, we propose using as thefixed a priori choice for the shape parameter of the GG distribu-tion. Note that we do not claim that is the right value forall types of images. Rather, we are saying that the estimation of

directly from the data is a complicated problem and in the ab-sence of accurate prior information on, provides strongrestoration results with guaranteed global convergence proper-ties.

The role of the parameter is two-fold. First it controls howclose the approximation in (6) is to the original norm. Using arelativelysmall providesbetter restorationofedges in the imagesince a smaller value provides better approximation to thenorm. Second, it essentially determines the convergence speedof the algorithm. While we do not intend to carry out a numericalanalysisofthefixedpointiterationin(9),thebasicreasonisthatfor

, in (10) isnotdifferentiableat and instability inthe numerical computations may arise. Ifis relatively large, thealgorithm is fast,and theconvergencespeeddeterioratesasgetssmaller. Therefore, should be set so as to achieve a compromisebetween the convergence speed and the edge preservation. Basedon our experience on natural scenes, we found that restorationsobtained for were visually indistinguishable from therestorations obtained for . We note that a similar value isrecommendedfor theTValgorithm[19].

V. REGULARIZATION PARAMETERS SELECTION

In this paper, we use a multidimensional extension of the-curve method [20], called the-hypersurface method [34], to

determine in (5). In order to describe the method thoroughly,we consider the following generalized image restoration schemewhere the estimate of the original imageis obtained by mini-mizing the following cost function

(11)

where are the regularization parametersand are the corresponding regularization operators. The costfunction in (11) represents a multiply constrained least squaresproblem and includes many popular image restoration schemesas its special cases. Our wavelet domain image restoration algo-rithm is obtained if , and are in the wavelet domain and

are the operators which extract desired por-tions of the wavelet transform of. For example, we can take

as the operator extracting the coarsest scale scaling coeffi-cients and as the operator extracting the wavelet coefficientsfor a doubly constrained, Model 1-type problem.

To extend the -curve, we first introduce the following quan-tities:

With the above definitions, the “-hypersurface” [34] is definedas a subset of associated with the map

, such that

For a single constraint, the-hypersurface reduces to the con-ventional -curve which is a plot of the residual norm versus


(a) (b) (c)

Fig. 2. (a)L-hypersurface, (b) Gaussian curvature of theL-hypersurface in (a), and (c) the norm of the difference between the actual and restored images.�

regularizes the coarsest scale scaling coefficients and� is used to penalize the wavelet coefficients in a Model 1 regularization scheme.

the norm of the restored image in a doubly logarithmic scale fora set of admissible regularization parameters. In this way, the

-curve displays the compromise between the minimization ofthese two quantities. It has been argued and numerically shownthat the so called “corner” of the-curve corresponds to a pointwhere regularization and perturbation errors are approximatelybalanced [20].

Analogous to the one dimensional case, the-hypersurfaceis a plot of the residual norm against the constraint norms

, . Intuitively, the “generalized corner” ofthe -hypersurface should correspond to a point where regu-larization errors and perturbation errors are approximately bal-anced. By a generalized corner, we mean a point on the surfacearound which the surface is maximally warped. We can mea-sure how much a surface is warped around a point by computingthe Gaussian curvature [34]. In Fig. 2, we plot a typical-hy-persurface along with its Gaussian curvature and the error be-tween the original and the restored images for a range of reg-ularization parameters. The experiment for which the-hyper-surface was computed was the restoration of a imagedegraded by a Gaussian blur of variance one pixel and corruptedby white Gaussian noise at 30 dB SNR. We used our multiscalealgorithm with and Model 1 regularization scheme asshown in Fig. 5(a). Fig. 2(b) shows the curvature of the-hy-persurface shown in Fig. 2(a) with (resp. ) being the regu-larization parameter for the scaling (resp. wavelet) coefficients.Fig. 2(b) and (c) clearly indicate the usefulness of the Gaussiancurvature plot in assessing the goodness of regularization pa-rameters. It is observed that the curvature is significant alongan extended maxima which is very close to a region in the errornorm, , plot in Fig. 2(c) where the error betweenthe actual and the restored images is minimized. Moreover, thecurvature plot indicates that there is in fact more than one goodregularization parameter for the scaling coefficients, and as longas we choose the correct value for the regularization parametercorresponding to the wavelet coefficients the restorations shouldhave approximately the same quality. The error norm plot inFig. 2(c) supports this point of view.

For the numerical experiments described in Section VI,the regularization parameters are selected by searching overa grid of parameter values in space and choosing that pointwhose curvature is maximum. The computational complexityof this technique is clearly quite high. The major difficulty

here is that the curvature of the-hypersurface possesses manymaxima/minima as seen in Fig. 5(c) and therefore the use ofwell-known optimization techniques such as the Gauss–Newtonmethod would fail. As the primary issue of interest here isin demonstrating that there is utility to the-hypersurfacemethod, we leave the considerable effort of finding a moreefficient implementation to future work.

VI. EXPERIMENTAL RESULTS

In this section, we illustrate the performance of our proposedmultiscale image restoration algorithm. All computationswere carried out by using the Matlab commercial softwarepackage with double precision arithmetic. We used the routinesin Donoho’s Wavelab toolbox [35] for the computation offorward and inverse wavelet transforms with Daubechies’eight tap most symmetrical wavelets [29]. In all cases below,we limited the number of levels of wavelet decomposition tothree. In the first example, we used a Gaussian convolutionalkernel, , with

to blur the mandrill image. Zeromean white Gaussian noise was added to set the SNR to 30 dB.In Fig. 3(a) and (b), we display the original and the blurred,noisy images.

We restored the degraded mandrill image using three reg-ularization techniques: our proposed multiscale regularizationscheme, the constrained least squares (CLS) algorithm with a2-D Laplacian regularizer [2], and the TV algorithm. The CLSand the TV algorithms are special cases of the generalized imagerestoration scheme in (11) for in which taking ,and (i.e., 2-D Laplacian) results in the CLS cost func-tion and and gives the cost function corre-sponding to the TV algorithm. The action of the 2-D Laplacianoperator at the pixel is

andthe action of the gradient operator is given by

withand . Both

operators are implemented by circulantly wrapping the image atthe boundaries. The relevant regularization parameters were de-termined using the -curve or the -hypersurface method. Forthe TV algorithm and our algorithm we used . Exper-imental results obtained for indicate that smaller


(a) (b)

(c) (d)

Fig. 3. (a) Original mandrill image, (b) blurred image, 30 dB SNR, (c) restored by the CLS algorithm, and (d) restored by the TV algorithm.

choices of do not improve the visual quality of the restora-tions [see Fig. 4(c) and (d) and Fig. 8(e) and (f)].

In Fig. 3(c) and (d) and Fig. 4(a)–(d) we display the restoredmandrill images corresponding to the CLS, the TV and the mul-tiscale algorithm. For our multiscale image restoration methodwe computed four restorations, displayed in Fig. 4(a)–(d), ac-cording to the Models 1 and 2 regularization schemes describedin Section III. Figs. 3 and 4 show that both the TV algorithm andour algorithm produce restored images visually superior to theCLS algorithm. We also observe that the images restored by ouralgorithm are a little sharper than the image restored by the TValgorithm and that the texture-like regions abundant in the man-drill image (e.g., the hairs around the mouth of the mandrill) arebetter recovered by our algorithm. The root mean square error(RMSE), , between the original and re-stored images are listed in Table I. For the Model 1 restorationin Fig. 4(a) the -hypersurface was used to determine two pa-

rameters, and corresponding to the coarsest scale scalingcoefficients and the wavelet coefficients respectively as shownin Fig. 5(a). In this case, the curvature of the-hypersurfaceis a 2-D function of the regularization parameters as seen inFig. 6(a). Also shown in Fig. 6(b) is a plot of RMSE as a func-tion of these regularization parameters. Examining these plotsshows that the curvature surface has a distinct extended maximaalong which the norm of the error is very close to being a min-imum. Thus, we see that the restoration algorithm is not overlysensitive to the scaling coefficient regularization parameter andlocating the correct regularization parameter for the wavelet co-efficients is more important.

In the Model 2 restoration in Fig. 4(b), each scale in thewavelet domain is assigned a different regularization parameteras seen in Fig. 5(b). Based on the-hypersurface obtained forthe Model 1 restoration in Fig. 6(a), we set the scaling coeffi-cient regularization parameter to . Fig. 7(a)–(c) shows the


(a) (b)

(c) (d)

Fig. 4. (a) Restored by the proposed algorithm using Model 1 regularization scheme withp = 1:0 and� = 1:2, (b) restored by the proposed algorithm usingModel 2 regularization scheme withp = 1:0, (c) Model 1 restoration with optimal� andp, and (d) Model 1 restoration with the same parameters as in (c) exceptthat� = 10 .

TABLE IRMSE VALUES FOR EACH

FIGURE

curvature of the -hypersurface obtained for this experiment.Since in this case the curvature is a three-dimensional function(one parameter for each wavelet scale), each of the 2-D plotsin Fig. 7(a)–(c) is actually a slice of the curvature hypersurface

with the regularization parameter corresponding to the coarsestscale being constant. Again, the maxima of the curvature of the

-hypersurface track well the minima of the RMSE surface sothat we are close to the “optimal” regularization parameters. Wesee little difference either in terms of the error norm or in termsof visual quality between the Models 1 and 2 restorations inFig. 4(a) and (b). This example verifies the primary assump-tion of Model 1 scheme where it was assumed that the varianceof the wavelet coefficients decrease uniformly across scales ac-cording to an exponential law.

Finally, in Fig. 4(c) and (d) we display the Model 1 restora-tions corresponding to an idealized case where the parametersand were estimated directly from the original image. Clearly,this is not a realistic situation since in practice the original imageis not available. Nonetheless, this example is interesting since it


Fig. 5. (a) Model 1, (b) Model 2, and (c) Model 3 regularization schemes as used in our experiments. In each model, the required regularization parameters,� ,are selected by theL-hypersurface method. In Models 1 and 3,� is set to 1.2a priori.

(a) (b) (c) (d)

(e) (f) (g) (h)

Fig. 6. Restoration of mandrill image. (a) Curvature of theL-hypersurface for Model 1 with� = 1:2 andp = 1:0, (b) the corresponding RMSE surface, (c)Curvature of theL-hypersurface for Model 1 with optimal� andp, (d) the corresponding RMSE surface, (e)L-curve for the CLS algorithm, (f) correspondingRMSE curve, (g) curvature of theL-curve for the TV algorithm, and (h) corresponding RMSE curve.

gives us an idea about how much improvement can be expectedwhen using the optimal and values as opposed to fixedapriori choices and . The optimal was es-timated by using the method proposed in [14] and was foundto be . The exponential parameter was es-timated by computing the slope of the line fitted to thefor . It was found to be .Since yields a nonconvex optimization task, we computedthe restorations for this case in two stages. The first stage startswith computing the restoration for , which is unique andthen the restored image for is fed as the starting pointto the restoration algorithm with . There is no guar-antee that the restored image for corresponds to the globalminimum of the cost function, nevertheless we obtained goodresults with this scheme. Fig. 6(c) and (d) shows the-surfaceand the RMSE surface for , respectively. Fig. 4(c) and (d)are the restorations obtained for this case. Fig. 4(c) and (d) dif-fers only in the value used for which was set to in (c) and

in (g). As already pointed out the two restorations arevisually indistinguishable, though the convergence of the algo-rithm took significantly longer for . Finally, compar-ison of Fig. 4(c) and (d) with Fig. 4(a) and (b) reveals that thereis visually little difference between the restored images corre-sponding to and cases. This example shows thatusing does not yield a significant improvement in the per-formance of the multiscale algorithm.

In our second example, we first blurred the original bridgeimage in Fig. 8(a) with a uniform motion blur and addedwhite Gaussian noise to the degraded image to set the SNRat 40 dB. The blurred image obtained by this way is shownin Fig. 8(b). Having established the edge preserving utility ofthe TV and the proposed algorithm over the conventional CLSmethod, we only display the restorations obtained by the TV andthe proposed algorithm in Fig. 8(c)–(f). For our multiscale algo-rithm, we applied the Models 1 and 2 regularization schemeswith the -hypersurface choice of regularization parameters.


(a) (b) (c)

(d) (e) (f)

Fig. 7. (a)–(c) Curvature of theL-hypersurface and (d)–(f) RMSE plots for the Model 2 restoration of the mandrill image.

(a) (b) (c)

(d) (e) (f)

Fig. 8. (a) Original bridge image, (b) blurred image, 40 dB SNR, (c) restored by the TV algorithm, (d) restored by the proposed algorithm using: (d) Model 1regularization scheme, (e) Model 2 regularization scheme, and (f) Model 2 restoration with the same parameters as in (e) but with� = 10 .

As in the previous example, we determined two regularizationparameters corresponding to the scaling and the wavelet co-efficients for Model 1 and three parameters corresponding tothe wavelet coefficients at each scale for Model 2. In Model 2restoration, the regularization parameter for the scaling coeffi-cients were set to from Fig. 9(a). Fig. 8(d) shows the Model1 restoration and Fig. 8(e) and (f) show the Model 2 restora-tions. In Fig. 8(e) and (f) all parameters except forare thesame [ in (e) and in (f)].

Although the RMSE values in Table I were similar, the re-stored images in Fig. 8(c) and (d) exhibit vastly different vi-sual characteristics. The TV algorithm fails to recover many ofthe small features in the image and produces an overly homog-enized restoration resembling an “oil painting” of the originalscene. The multiscale algorithm is able to reproduce finer detailthereby yielding a more visually appealing restoration. As in theprevious example, we see little difference in terms of the visualquality between the Models 1 and 2 restorations. Note that we


(a) (b)

(c) (d)

Fig. 9. Restoration of the bridge image. (a) Curvature of theL-hypersurfacefor the proposed algorithm, (b) curvature of theL-curve for the TV algorithm,and (c)-(d) corresponding RMSE plots.

used the same value in both mandrill and bridge exam-ples regardless of the image considered.

In our final example, we demonstrate the orientation adap-tive nature of our approach. In Fig. 10(a), we display an artifi-cial image which has significant structure in the hori-zontal direction, but little in the vertical and diagonal directions.This image was blurred by a Gaussian convolutional kernel with

, and zero mean white Gaussian noise was added toset the SNR at 30 dB. Because of the large differences betweenthe structure in the horizontal and vertical directions, an idealimage restoration algorithm should use different regularizationparameters for vertical, horizontal and diagonal directions. Withthis in mind, in Fig. 10(c) and (d) we display the restorations ob-tained using Models 1 and 3 schemes which require three regu-larization parameters, , , and , as displayed in Fig. 5(b)and (c), respectively. The-hypersurface was employed to de-termine the required regularization parameters. For both Models1 and 3 schemes, we set the scaling coefficient regularization pa-rameter to . For the Model 3 restoration, the regularizationparameters obtained for the vertical and diagonal orientations(in which the image is constant) were approximately two ordersof magnitude larger than the regularization parameter obtainedfor the horizontal orientation. It is clear from Fig. 10(c) and (d)that the orientation adaptive algorithm produces a much betterrestoration than the scale adaptive algorithm.

VII. CONCLUSION

In this paper, we introduced a wavelet domain multiscaleimage restoration algorithm for use in linear image restorationproblems. Following the recent results in the area of image de-noising and coding, we developed a statistical prior model forthe wavelet coefficients of images. Our priors are able to cap-ture spatial, scale and orientational characteristics of imagesaccurately. We developed a half-quadratic algorithm to solvethe nonlinear optimization problem resulting from using such

(a) (b)

(c) (d)

Fig. 10. (a) Original image, (b) blurred image, 30 dB SNR, (c) restoredby the proposed algorithm with Model 2 (scale adaptive) regularization, and(d) restored by the proposed algorithm with Model 3 (orientation adaptive)regularization.

priors and utilized the -hypersurface method for choosing therequired regularization parameters. Experimental results showthat our algorithm can produce restorations which are visuallysignificantly better than that of the traditional techniques andat least comparable, if not better, than that of the the edge-pre-serving algorithms.

ACKNOWLEDGMENT

The authors would like to thank the anonymous reviewersfor their comments and suggestions which greatly improved thequality of this paper

REFERENCES

[1] R. L. Lagendijk and J. Biemond,Iterative Identification and Restorationof Images. Norwell, MA: Kluwer, 1991.

[2] H. C. Andrews and B. R. Hunt,Digital Image Restoration. EnglewoodCliffs, NJ: Prentice-Hall, 1977.

[3] D. Geman and C. Yang, “Nonlinear image recovery with half-quadraticregularization,”IEEE Trans. Image Processing, vol. 4, pp. 932–946, July1995.

[4] P. Charbonnier, G. Aubert, L. Blanc-Feraud, and M. Barlaud, “Sto-chastic relaxation, Gibbs distribution, and the Bayesian restoration ofimages,” IEEE Trans. Image Processing, vol. 6, pp. 298–311, Feb.1997.

[5] S. Osher, L. I. Rudin, and E. Fatemi, “Nonlinear total variation basednoise removal algorithms,”Phys. D, vol. 60, pp. 259–268, 1992.

[6] R. W. Buccigrossi and E. P. Simoncelli, “Image compression via jointstatistical characterization in the wavelet domain,” GRASP Lab., Univ.Pennsylvania, Philadelphia, Tech. Rep. 414, May 1997.

[7] E. P. Simoncelli and E. Adelson, “Noise removal via Bayesian waveletcoring,” in Proc. 1996 IEEE Int. Conf. Image Processing, vol. 1, Lau-sanne, Switzerland, Sept. 1996, pp. 379–382.

[8] A. Chambolle, R. E. DeVore, N. Lee, and B. J. Lucier, “Nonlinearwavelet image processing: Variational problems, compression, andnoise removal through wavelet shrinkage,”IEEE Trans. Image Pro-cessing, vol. 7, pp. 320–335, Mar. 1998.


[9] E. L. Miller and A. S. Willsky, “A multiscale approach to sensor fusionand the solution of linear inverse problems,”Appl. Comput. Harmon.Anal., vol. 2, pp. 127–147, 1995.

[10] , “Multiscale, statistically-based inversion scheme for the linearizedinverse scattering problem,”IEEE Trans. Geosci. Remote Sensing, vol.34, pp. 346–357, Mar. 1996.

[11] M. R. Banham and A. K. Katsaggelos, “Spatially adaptive wavelet-basedmultiscale image restoration,”IEEE Trans. Image Processing, vol. 5, pp.619–633, Apr. 1996.

[12] G. Wang, J. Zhang, and G. W. Pan, “Solution of inverse problemsin image processing by wavelet expansions,”IEEE Trans. ImageProcessing, vol. 4, pp. 579–593, May 1995.

[13] P. Mathieu, M. Antonini, M. Barlaud, and I. Daubechies, “Image codingusing wavelet transform,”IEEE Trans. Image Processing, vol. 1, pp.205–220, Feb. 1992.

[14] S. Mallat, “A theory for multiresolution signal decomposition: Thewavelet representation,”IEEE Trans. Pattern Anal. Machine Intell.,vol. 11, pp. 674–693, July 1989.

[15] Y. Meyer,Wavelets and Operators. New York: Cambridge Univ. Press,1992.

[16] R. A. DeVore, B. Jawerth, and B. J. Lucier, “Image compression throughwavelet transform coding,”IEEE Trans. Inform. Theory, vol. 38, pp.719–747, Mar. 1992.

[17] D. T. Vuza and U. Amato, “Besov regularization, thresholding andwavelets for smoothing data,”Numer. Funct. Anal. Optim., vol. 18, no.5/6, pp. 461–493, 1997.

[18] D. L. Donoho, “Unconditional bases are optimal bases for data compres-sion and for statistical estimation,”Appl. Comput. Harmon. Anal., vol.1, pp. 100–115, Dec. 1993.

[19] C. R. Vogel and M. E. Oman, “Fast, robust total variation-based recon-struction of noisy, blurred images,”IEEE Trans. Image Processing, vol.7, pp. 813–824, July 1998.

[20] P. C. Hansen, “Analysis of discrete ill-posed problems by means of theL-curve,” SIAM Rev., vol. 34, pp. 561–580, 1992.

[21] C. R. Vogel, “Non-convergence of the L-curve regularization parameterselection method,”Inverse Problems, vol. 12, no. 4, pp. 535–547, 1996.

[22] M. Hanke, “Limitations of the L-curve method in ill-posed problems,”BIT, vol. 36, no. 2, pp. 287–301, 1996.

[23] A. Tikhonov and V. Arsenin,Solution of Ill-Posed Problems. NewYork: Wiley, 1977.

[24] M. Bertero, T. A. Poggio, and V. Torre, “Ill-posed problems in earlyvision,” Proc. IEEE., vol. 76, pp. 869–889, Aug. 1988.

[25] M. Bertero, “Regularization methods for linear inverse problems,”In-verse Problems, vol. 1225, pp. 52–112, 1986.

[26] H. W. Engl, M. Hanke, and A. Neubauer, “Regularization of inverseproblems,” in Mathematics and Its Applications. Dordrecht, TheNetherlands: Kluwer, 1996.

[27] G. Archer and D. M. Titterington, “On some Bayesian regularizationmethods for image restoration,”IEEE Trans. Image Processing, vol. 4,pp. 989–995, July 1995.

[28] J. O. Berger,Statistical Decision Theory and Bayesian Analysis. NewYork: Springer-Verlag, 1985.

[29] I. Daubechies,Ten Lectures on Wavelets. New York: SIAM, 1992.[30] R. Dufour and E. L. Miller, “Statistical signal restoration with wavelet

domain prior models,”Signal Process., vol. 78, pp. 289–307, 1999.[31] G. H. Golub and C. E. Van Loan,Matrix Computations, 2nd

ed. Baltimore, MD: John Hopkins Univ. Press, 1989.[32] M. E. Zervakis, T. M. Kwon, and J.-S Yang, “Multiresolution image

restoration in the wavelet domain,”IEEE Trans. Circuits Syst., vol. 42,pp. 578–591, Sept. 1995.

[33] P. Moulin and J. Liu, “Analysis of multiresolution image denoisingschemes using generalized-gaussian and complexity priors,”IEEETrans. Inform. Theory, vol. 38, pp. 719–747, Apr. 1998.

[34] M. Belge, M. E. Kilmer, and E. L. Miller, “Simultaneous multiple regu-larization parameter selection by means of the L-hypersurface with ap-plications to linear inverse problems posed in the wavelet domain,” inProc. SPIE’98: Bayesian Inference Inverse Problems, July 1998.

[35] J. Buckheit and D. Donoho, “Wavelab and reproducible research,” inWavelets and Statistics, A. Antoniadis and G. Oppenheim, Eds. Berlin,Germany: Springer-Verlag, 1995, pp. 55–81.

Murat Belge received the B.S.(Hon.) andM.S.(Hon.) degrees in electrical engineering fromBilkent University, Ankara, Turkey, in 1993 and1995, repectively, and the Ph.D. degree in electricaland computer engineering from NortheasternUniversity, Boston, MA, in September 1999.

He is currently a DSP Engineer with Aware, Inc.,Bedford, MA. His research interests include digitalsignal and image processing in general and imagerestoration and reconstruction in particular, telecom-munications and adaptive filtering.

Misha E. Kilmer received both the B.S. (magna cumlaude) and M.A. degrees in mathematics from WakeForest University, Winston-Salem, NC, in 1992 and1994, respectively, and the Ph.D. degree in appliedmathematics from the University of Maryland, Col-lege Park, in December 1997.

She is currently an Assistant Professor of math-ematics at Tufts University, Medford, MA. Her re-search interests include preconditioning and iterativemethods for large-scale scientific computation, nu-merical analysis, numerical linear algebra, and image

processing.Dr. Kilmer was the recipient of a 1997 Society for Industrial and Applied

Mathematics (SIAM) Student Paper Prize award and has also received studentawards for conference travel in the U.S. and Europe. She is a member of PhiBeta Kappa and is a member of SIAM, AWM, AMS, and MAA.

Eric L. Miller (S’90–M’95) received the S.B., S.M.,and Ph.D. degrees in electrical engineering andcomputer science from the Massachusetts Instituteof Technology, Cambridge in 1990, 1992, and 1994,respectively.

He is currently an Assistant Professor with theDepartment of Electrical and Computer Engineering,Northeastern University, Boston, MA. His researchinterests include the exploration of theroretical andpractical issues surrounding the use of multiscaleand statistical methods for the solution of inverse

problems in general and inverse scattering problems in particular, and thedevelopment of computationally efficient, physically based models for use insignal processing applications.

Dr. Miller is a member of Tau Beta Pi, Eta Kappa Nu, and Phi Beta Kappa andreceived the CAREER Award from the National Science Foundation in 1996.He is currently serving as an Associate Editor for the IEEE TRANSACTIONS ON

IMAGE PROCESSING.

wavelet domain image restoration with adaptive edge-preserving regularization

Documents