space-varying restoration of optical images

3162 J. Opt. Soc. Am. A/Vol. 14, No. 12 /December 1997 Nagy et al.

Space-varying restoration of optical images

James G. Nagy

Department of Mathematics, Southern Methodist University, Dallas, Texas 75275

V. Paul Pauca

Department of Computer Science, Duke University, Durham, North Carolina 27708

Robert J. Plemmons and Todd C. Torgersen

Department of Mathematics and Computer Science, Wake Forest University, Winston-Salem, North Carolina 27109

Received August 22, 1996; accepted July 21, 1997; revised manuscript received July 25, 1997

The improvement in optical image quality is now generally attempted in two stages. The first stage involvestechniques in adaptive optics and occurs as the observed image is initially formed. The second stage of en-hancing the quality of optical images generally occurs off line and consists of the postprocessing step of imagerestoration. Image restoration is an ill-posed inverse problem that involves the removal or the minimizationof degradations caused by noise and blur in an image, resulting from, in this case, imaging through a medium.Our work concerns a new space-varying regularization approach and associated techniques for accelerating theconvergence of iterative image postprocessing computations. Denoising methods, including total variationminimization, followed by segmentation-based preconditioning methods for minimum residual conjugate gra-dient iterations, are investigated. Regularization is accomplished by segmenting the image into (smooth) seg-ments and varying the preconditioners across the segments. The method appears to work especially well onimages that are piecewise smooth. Our algorithm has computational complexity of only O(ln2 log n), wheren2 is the number of pixels in the image and l is the number of segments used. Also, parallelization is straight-forward. Numerical tests are reported on both simulated and actual atmospheric imaging problems. Com-parisons are made with the case where segmentation is not used. It is found that our approach is especiallyattractive for restoring images with low signal-to-noise ratios, and that magnification of noise is effectivelysuppressed in the iterations, leading to a numerically efficient and robust regularized iterative restoration al-gorithm. © 1997 Optical Society of America [S0740-3232(97)02112-1]

1. INTRODUCTIONResearchers are actively seeking to overcome the degra-dation of optical image quality caused by the effects of im-aging through a medium such as air, liquids, and otherimage degradation media. For example, atmosphericturbulence has especially frustrated astronomers sincetelescopes were invented. The optical effects are in partdue to the mixing of warm and cold atmospheric layers,resulting in nonuniformities in the density and the refrac-tive index of air. These variations cause parts of thelight waveforms from an object to be slowed by differentamounts, distorting the image. The resulting twinklingof the stars and other effects are the main limitations ofimaging through the atmosphere.1

Modern methods for the improvement in optical imagequality are often attempted in two steps. The first stepoccurs as the observed image is initially formed. Spe-cially designed deformable mirrors operating in a closed-loop adaptive-optics system can partially compensate forthe effects of medium turbulence. Optical systems detectthe distortions. A wave-front sensor measures theturbulence-induced wave-front distortions by using lightfrom either a natural guide star or a guide star artificiallygenerated with the use of range-gated laser backscatter.These distortions are at least partially nullified by adjust-ing the surface shape of the deformable mirror. To be ef-

0740-3232/97/123162-13$10.00 ©

fective, these corrections have to be performed at real-time speed. Adaptive-optics control systems form thesubject of considerable recent investigation (see, e.g.,Ellerbroek et al.2,3 and Hardy4).

The second stage of compensating for the effects of im-aging through a medium occurs generally off line and con-sists of the postprocessing step of image reconstruction orrestoration. Here, large-scale computations, using eithera simultaneous image of a natural guide star or a largeensemble of images corresponding to different atmo-spheric realizations, are used to deconvolve the blurringeffects of atmospheric turbulence (e.g., Lagendijk andBiedmond5 and Nagy et al.6,7). Our concern in this paperis the development of robust regularized accelerationtechniques for large-scale image postprocessing itera-tions. Adaptive linear and nonlinear methods are con-sidered.

There are a number of applications for optical imagepostprocessing:

• Ground-based atmospheric imaging. Here, resto-ration procedures are used to deconvolve the blurring ef-fects of atmospheric turbulence.

• Endoscopy. Images are obtained throughaberration-inducing fluids, and thus optical image resto-ration can often improve the image quality.

1997 Optical Society of America

Nagy et al. Vol. 14, No. 12 /December 1997 /J. Opt. Soc. Am. A 3163

• Microscopy. This involves confocal, fluorescence,and scanning microscopy, sometimes in three dimensions,producing images that often must be restored by postpro-cessing.

• Underwater imaging. Again this technology isknown to be limited by turbulence and turbidity.

The outline of our paper follows. A review of relevantbackground material on image denoising and image de-blurring methods is given in Section 2. A space-varyingregularized iterative restoration method based on imagesegmentation is proposed in Section 3. Numerical testson simulated and real atmospheric imaging data are re-ported in Section 4. Some observations and directions forfuture work are discussed in Section 5.

2. IMAGE POSTPROCESSINGLinear image postprocessing enhancement methods, es-pecially direct fast-Fourier-transform- (FFT-) based de-convolution and related approaches, are currently used inmany practical applications because of their simplicityand speed. This is especially true in medical imaging,where even the use of iterative methods8–10 has not yetbeen widely accepted in practice (see, e.g., Biedmondet al.8). Convolution backprojection and related directfiltering schemes are often used as standard approachesto image reconstruction and restoration. Two main prob-lems with standard linear methods are oscillations andsmoothing. Traditional enhancement methods preventoscillations in the restored image by smooth regulariza-tion techniques.11,12 Here, discontinuous or singular im-age features and oscillatory textures sometimes cannot berestored. We focus on overcoming these restrictions byusing space-varying regularization methods. We alsoprovide methods for reducing the computational require-ments of certain iterative image enhancement schemes tomake their use more practical for applications where bothenhancement quality and speed are important consider-ations.

If we let H denote the blurring operator and h thenoise process, then the image restoration problem withadditive noise can be expressed as the linear operatorequation

g 5 H f 1 h, (1)

where g and the unknown f denote functions containingthe information of the recorded and original images, re-spectively. Note that when H 5 I , the identity opera-tor, the image restoration problem reduces to extractingthe image f from a noisy image g. This problem is usu-ally referred to as the denoising problem.

Let u and v denote two-dimensional variables in, say, arectangular domain V in R2. If H is a convolution op-erator, as is often the case in optical imaging, then the op-erator acts uniformly (i.e., in a spatially invariant man-ner) on f defined on V. Here, Eq. (1) can be written as

H f ~u ! 5 EV

h~u 2 v !f~v !dv. (2)

The problem is to both deconvolve and denoise the re-corded image during the reconstruction process, and we

refer to this as the denoising and deblurring problem. Inoptical imaging the kernel h in Eq. (2) is called the con-volution point spread function (PSF). After discretiza-tion of Eq. (1), the spatial operator H defined by h in Eq.(2) is a matrix that we denote by H. Here, in the spa-tially invariant case, H is a block Toeplitz matrix withToeplitz blocks.5

A classical approach employed for solving Eq. (1) is thatof penalized least squares, which is also called Tikhonovregularization in the inverse problems literature. Thisrequires minimization of the expression

iH f 2 gi2 1 aJ~ f !, (3)

where i•i denotes the norm on L2(V), a is a positive(regularization) parameter, and the functional J( f )serves the purpose of stabilizing the least-squares prob-lem and penalizing certain undesirable artifacts like spu-rious oscillations in the computed f. Various choices ofJ( f ) can be made, including iSf i2, where S is somesmoothing differential operator, or the identity. Thismodel leads to fast linear methods for computing f and isoften the method of choice of practitioners.5 However,the use of other norms, such as the L1 norm, leads to non-linear minimization methods,13 which sometimes resultin superior enhancement of blocky, noisy images, but withadded computational cost. Such approaches are brieflyreviewed in the next subsection.

A. Variational MethodsOsher and Rudin14 have suggested an image enhance-ment method based on solving a nonlinear partial differ-ential equation constrained minimization problem, wherethe function being minimized is the total variation (TV) ofthe image f 5 f (x, y). Numerous papers on various TVapproaches for image denoising or restoration have beenwritten in the past five years. A partial list is providedin the references.13–19 Recent papers have concentratedon speeding up the computations. Significant computa-tional requirements are often necessary for these nonlin-ear TV-based methods.

The variational method of Rudin and co-workers17,18

considers the following constrained minimization prob-lem:

minf

EV

u¹f udu subject to iH f 2 gi2 5 s2, (4)

where ¹f denotes the gradient of f and s is the noise level.At a point u 5 (x, y) in the image domain, f(u)5 f(x, y), and so

u¹f ~u !u 5 Afx2 1 fy

2. (5)

The quantity

EV

u¹f udu 5 EV

@ fx2~x, y ! 1 fy

2~x, y !#1/2 dxdy (6)

is called the TV norm of f. The minimization in expres-sion (4) is a form of regularization, a step necessary insolving most ill-posed inverse problems. The TVmethod17,18 is especially effective for recovering a blocky,discontinuous function from noisy data.16


Several authors13,15,16,19 have considered the followingclosely related Tikhonov regularization problem [expres-sion (3)], where

aJ~ f ! 5 aEV

u¹f udu (7)

(see, e.g., Acar and Vogel15 and Vogel and Oman19). Herea is a positive regularization parameter that measuresthe trade-off between a good fit and an oscillatory solu-tion. This method corresponds to the use of the L1 normin the discrete case.

Various numerical schemes have been devised to obtainthe minimizer of the functional (7). For example, in Ru-din and Osher17 and in Rudin et al.,18 an explicit time-marching scheme is used. However, the time step mustbe chosen small. Thus the number of iterations to opti-mal convergence can be quite large.13 Vogel and Oman19

have introduced a lagged diffusivity fixed-point iterationapproach, to perform the TV computations. In the de-noising case, numerical experiments cited in Vogel andOman19 indicated that the lagged diffusivity fixed-pointiteration method often gives a faster convergence ratethan the time-marching method, with overall greaterspeed for the entire process. We use their approach inour work.

A disadvantage of experimentally obtained data setsrepresenting the PSF’s (obtained, for example, by usingguide stars) is that they are also subject to degradationscaused by noise during the image formation process.Thus removal of such degradations may be necessary be-fore any computations using the PSF in the image resto-ration postprocessing step. In this paper we consider,among other approaches, TV-based denoising of the PSFwhose model is formulated as

s 5 h 1 h, (8)

where h is the blur produced by the atmospheric turbu-lence on a single point and s is the actual measurement,or the observed PSF.

In our experiments it has been found that TV denoisingof this noisy PSF in a preprocessing step improves thespace-varying restoration (SVR) process to be described inSection 3. This observation holds even though the PSFis, in this case, not a blocky image. Also, TV denoising ofthe PSF can be accomplished with little extra computa-tion, since the PSF’s h for optical imaging can usually betreated as having small extent.1 The cost of preprocess-ing the PSF is much less than that of deblurring f. Ourspace-varying approach to the deblurring step is consid-ered in the following sections.

B. Preconditioned Iterative Regularization MethodWe now consider numerical methods for approximatingthe solution to the linear restoration problem in dis-cretized (matrix) form obtained from the operator equa-tion (1):

g 5 Hf 1 h. (9)

For simplicity we assume that the image in question isn 3 n, and accordingly H is n2 3 n2.

There are many ways to handle this inverse problem ofrecovering f, including direct methods such as the singu-lar value decomposition and linear iterative methods suchas conjugate gradients (CG) or the nonlinear TV methodsdescribed in Subsection 2.A. Direct methods can some-times be impractical for problems having matrices oflarge dimensions, as is the case in image restoration.Here, we consider iterative methods. We must also con-sider the fact that Eq. (9) is an ill-posed inverse problem.That is, a computed solution to Hf 5 g is likely to be ex-tremely corrupted by noise. Regularization methods at-tempt to alleviate sensitivity to the noise by filtering outeigencomponents of the solution belonging to the noisesubspace. For some iterative methods, such as CG, ithas been established that early termination of the itera-tions accomplishes this regularization effect.11,20

It is known that convergence of the CG method will befast if the singular values of H are clustered around 1,21

which can often be accomplished through preconditioning.Preconditioning amounts to constructing a matrix C suchthat HC21 has a more favorable spectrum. The CGmethod is then used to solve HC21y 5 g, with the solu-tion f given by Cf 5 y.

Preconditioning of ill-posed inverse problems is a deli-cate matter. Although clustering of the spectrum willprovide fast convergence rates, it can also have the disad-vantage of mixing the signal and noise subspaces.22

Thus an effective preconditioner for ill-posed inverseproblems should cluster only the spectrum belonging tothe signal subspace. Moreover, it must be relatively easyto solve systems of equations with C.

The classical CG method requires that H be symmetricand positive definite. Although the theoretical model ofthe blurring operator is symmetric (e.g., a Gaussian PSFin atmospheric blurring), in optical imaging applications,H is often obtained from observations degraded by noiseand is not likely to retain a symmetric structure. Sincethe theoretical blurring model is symmetric, though, wewould expect that H is nearly symmetric and that onecould obtain a good symmetric approximation to the op-erator. This is the approach taken by Hanke and Nagy23

for spatially invariant blurs in astronomical imaging.Thus, without loss of generality, in the following we as-

sume that H is symmetric (if not, we symmetrize it by us-ing the techniques suggested by Hanke and Nagy23).However, because of noise and approximation errors, wecannot expect H to be positive definite. Therefore theclassical CG method is still not applicable. Althoughthere are many variants of the CG method that take ad-vantage of the symmetry of H,21 the question of whetherthey still retain a regularization effect through early ter-mination has, until recently, been unanswered. In par-ticular, an important result in this direction was estab-lished by Hanke24 for a minimum-residual- (MR-) typevariant of the CG method, which he refers to as MR-II.The MR-II method is a mathematically, but not computa-tionally, equivalent formulation of a method first pro-posed by Paige and Saunders.25

Hanke and Nagy23 used MR-II for restoration of atmo-spherically blurred images, assuming a spatially invari-ant PSF. Each iteration needs only one matrix–vectormultiplication and one preconditioner solve. The precon-


ditioning scheme can be briefly described as follows.Since H is a block Toeplitz matrix with Toeplitz blocks, itcan be approximated by a block circulant matrix with cir-culant blocks.26 Then C can be easily diagonalized bythe two-dimensional unitary FFT (Ref. 27); that is, C5 F* LF, where L is a diagonal and generally complexmatrix. With the use of FFT’s, solutions of linear sys-tems with C can be computed by using only O(n2 log n)arithmetic operations. Moreover, since the eigenvaluesof C are readily available, we can replace those with smallabsolute value, i.e., those associated with the noise sub-space, by 1. In this way C clusters the singular values ofH associated with the signal subspace and acts like theidentity on the noise subspace; see Ref. 22 for further de-tails.

In Hanke and Nagy’s work,23 the spectra of C associ-ated with the signal and noise subspaces are obtained byusing the L-curve criterion, as it is used to determine atruncation index for the truncated singular value decom-position. Specifically, given a truncation parameter t asdefined in Ref. 22, let

Fig. 1. Algorithm 1: MR-II with approximate inverse precon-ditioning by Qt .

zt 5 (ul ju.t

vj* g

l jvj , rt 5 g 2 Czt , (10)

where vj is the jth column of F* and l j is the correspond-ing eigenvalue in L. Then the L curve consists of a log–log plot of

i zt i2 versus irt i2. (11)

For larger t the solutions zt have a norm typically of thesame order as that of f, whereas the residual tends to de-crease. Thus this part of the plot remains relatively flat.As t is decreased, more components of the noise subspacebegin to corrupt zt , so that i zt i begins to grow large.On the other hand, irt i begins to approach the noiselevel, where it essentially remains constant as t is fur-ther decreased. Therefore the plot begins to rise at somepoint and thus has a distinct L-shaped appearance. Thevalue of t corresponding to the corner of the L typicallyprovides a reasonable estimate of the separation of thesignal and noise subspaces. That is, the eigencompo-nents corresponding to those l j satisfying ul ju . t are as-sociated with the signal subspace, and the remainingwith the noise subspace. For further details on the prop-erties of the L curve, we refer to Hansen and O’Leary28

and the references therein.Once a truncation parameter t is chosen, a precondi-

tioner Ct is defined as follows. Let

@L~t!# jj 5 H L jj if uL jju . t

1 otherwise, (12)

and take

Ct 5 F* L~t!F. (13)

The preconditioned MR-II (PMR-II) method is given as al-gorithm 1 in Fig. 1.

3. SPACE-VARYING RESTORATIONIn this section we introduce our space-varying restoration(SVR) approach for the purpose of deblurring optical im-ages. SVR is a variant of the preconditioned regulariza-tion MR-II method discussed in Subsection 2.B. It pro-vides a space-varying regularized iterative restorationscheme based on image segmentation. Recently, otherimage restoration methods based on image segmentationhave been investigated.29–31

The motivation for our work is based on the observa-tion that the preconditioned regularization MR-II methodof Hanke and Nagy23 applies the same level of regulariza-tion and preconditioning to the entire image, regardless ofhow much information or noise is present in various por-tions of the image. Note that in regions containing lowersignal-to-noise ratios (SNR’s) (such as background areas),one would expect fast convergence to amplify noise; thatis, there may be too much preconditioning in these re-gions. Therefore we propose to apply regularized precon-ditioning in a space-varying manner. Specifically, ourapproach considers a multilevel preconditioning schemein order to vary the level of restoration, depending on theamount of activity in various regions of the image. Theseactivity regions are identified by a segmentation scheme


described in the Subsection 3.A. Once these regions aredefined, we use the L-curve scheme28 on each region toidentify the signal and noise subspaces and hence thelevel of restoration. Thus we expect that more regular-ized preconditioning will be used in regions of high activ-ity and less regularized preconditioning in regions of lowactivity. The method should work especially well on im-ages that are piecewise smooth.

For the image data presented in Section 4, the bound-aries of the activity regions tend to coincide with steepedges in the image; thus the segmentation scheme also re-sults in essentially smooth segments. The net effect is tosuppress the amplification of the noise signal by adap-tively accelerating convergence in areas of high SNR. Byregularizing the computation on independent smooth seg-ments, we achieve a stable SVR method.

A. Image SegmentationA simple approach to obtaining a fast segmentation forour optical image restoration purposes is the technique ofmultiple thresholding.32 Let the matrix G denote an ob-served n 3 n image of pixel values, and let g denote then2 vector obtained by unstacking G row-wise. The imageG is segmented into l nonoverlapping regions G1 ,G2 , ..., Gl by using a monotonically increasing sequenceof threshold values t0 , t1 , t2 , ..., tl and the followingrule: pixel gi is in segment Gm , for 1 < m < l, if andonly if tm21 , gi < tm .

It was determined that for the data used in our tests,some form of smoothing or denoising is necessary to im-prove the effectiveness of the segmentation procedure.Here, median filtering was chosen to smooth the imagesand reduce noise before segmentation by multiple thresh-olding. For all of the image data presented in Section 4,median filtering is used to smooth the observed imageand reduce noise before segmentation. For the restora-tions shown, threshold values t0 , ..., tl are chosen by in-spection. The threshold values chosen by inspectionwere found to be consistent with histogram-basedtechniques,32 which separate multimodalities in a histo-gram of the pixel values. Figure 2 illustrates the seg-

mentation scheme described above as applied to some realdata (see example 3 in Section 4).

B. Space-Varying Restoration AlgorithmWe now describe our scheme for extending and improvingupon the method of Hanke and Nagy by incorporatingsegmentation and SVR in different regions according toactivity level. As mentioned above, the basic idea is touse a slightly different preconditioner over each of the re-gions, so that convergence is accelerated in regions ofhigh activity.

Consider again matrix C, a block circulant with circu-lant blocks that is an approximation to H, and recall thatC 5 F* LF, where L is a diagonal matrix containing theeigenvalues of C, and F is the two-dimensional unitaryFT matrix. Given the nonoverlapping segmented regionsG1 , ..., Gl of the n 3 n observed image G, the L-curvecriterion suggested in Subsection 2.B can be used tochoose a truncation tolerance t i for the signal and noisesubspaces of each region. Let g be the vector obtainedfrom G by unstacking the image pixels rowwise. Thenthe segmentation of G can be equivalently defined as

g 5 D1 g 1 D2 g 1 ••• 1 Dl g, (14)

where Di is an n2 3 n2 diagonal masking matrix given by

@Di#kk 5 H 1 if pixel gk belongs to segment G ~i !

0 otherwise.

(15)

Then an approximate inverse preconditioner for regionG (i) is defined to be Qti

[ Ct21, where Ct is given in Eq.

(13), replacing t by t i .With the notation in place, we can now introduce our

SVR scheme. The SVR method proposed here is shownas algorithm 2 in Fig. 3. SVR is a MR-type method thatbelongs to a class of Krylov subspace methods24 and, morespecifically, is closely related to PMR-II. Using l seg-ments, SVR solves the problem Hf 5 g with n2 un-knowns by computing a projection of a sequence of im-plicit iterates of dimension ln2 produced by (modified)PMR-II applied to a larger, related problem. To under-

Fig. 2. Segmentation applied to satellite data: (a) image after 4 3 4 median filter, (b) segmentation into three regions.


stand the relation between SVR and PMR-II, consider theblock-diagonal approximate inverse preconditioned sys-tem

QTx 5 Qy, (16)

where the unknown x P R ln2and

Q 5 diag~Qt1, Qt2

, ..., Qtl!,

T 5 diag~H, H, ..., H !,

y 5 @ gT, gT, ..., gT #T. (17)

Note that the block-diagonal matrix Q is symmetric posi-tive definite, and Eq. (16) is equivalent to the system

Q1/2TQ1/2z 5 Q1/2y, x 5 Q1/2z. (18)

A key observation in the understanding of SVR in algo-rithm 2 is that the quantities wk

(i) in the main loop of thealgorithm are of the form (QT)kw0 , where w0 is formedby stacking the w0

(i) for 1 < i < l into a vector of dimen-sion ln2.

Fig. 3. Algorithm 2: SVR.

Further, let D be an n2 3 ln2 matrix representing aprojection D: R ln2 → R n2

given by

D 5 @D1 , D2 , ..., Dl#, (19)

where Di are the diagonal matrices defined in Eq. (15) for1 < i < l. Observe that an approximate restoration fkcomputed by SVR is a projection under D of a vector xkbelonging to a Krylov subspace given by

K k~r1 ; QT !

5 span@r1 , ~QT !r1 , ~QT !2r1 , ..., ~QT !kr1#, (20)

where r1 5 Q(y 2 Tx0) is the initial (preconditioned) re-sidual and x0 is formed from l repetitions of the initialguess f0 . The careful reader may observe that SVR canbe derived by substituting Q, T, x0 , and y into PMR-IIand making the two following modifications.

First, observe that the scalar quantities r, a, and b inSVR are computed similarly to the corresponding scalarquantities in PMR-II, except that the inner product usedin SVR is defined relative to the projection D. More pre-cisely, let ^ • , • &D denote a semidefinite inner product ona, b P R ln2

given by

^a, b&D 5 ^Da, Db&. (21)

See Hanke24 for a discussion of choice of norms and thecorresponding derivation of various CG-type methods. Itis possible that, for general a, b P R ln2

, the parameterb 5 ^wk , uk&D 5 ^Dwk , Duk& may become zero. Al-though at this writing it is not resolved whether b couldbecome zero for our particular applications, we emphasizethat this situation seems unlikely because of the repeti-tive way in which our vectors wk and uk of size ln2 areformed.

Second, the computation and the storage of uninterest-ing quantities are suppressed; e.g., an approximate solu-tion (restoration) fk requires only n2 values. The ap-proximated solution fk is obtained by projecting xk at eachiteration onto R n2

by using D; that is, fk 5 Dxk . Notealso that SVR reduces to PMR-II, that is, SVR computesexactly the same sequence of vector iterates as that byPMR-II when t1 5 t2 5 ••• 5 t l .

A number of variants of SVR are possible based on dif-ferent choices of which variables in PMR-II are projectedand which are fully represented in a space of dimensionln2. At one extreme all variables in PMR-II could befully represented in R ln2

. Here, this variant of SVR isequivalent to applying MR-II to the preconditioned sys-tem QTx 5 Qy and then combining the desired portionsof the independently computed solutions by using the di-agonal matrices Di . At the other extreme, all of the vari-ables in PMR-II could be represented by their projections;i.e., a solution to the system DQTx 5 DQy is sought.The SVR variant presented as algorithm 2 in Fig. 3 takesa position between these two extremes. Projected valuesare used for the purposes of computing the scalars r, a,and b.

Development of SVR variants remains a topic for fu-ture investigation. Formal convergence analysis of SVRwill also be presented in future work. At this point itseems that our restoration scheme as presented above


Fig. 4. Block image data: (a) true image, (b) PSF, (c) degraded image.

could be slightly modified to deal with nonuniform or spa-tially varying blur. Also, we note that SVR could be usedfor multiple-frame image restoration of an object. Bymultiple-frame restoration is meant the restoration of animage from a set of observations of both the PSF and theobserved image. This particular application is of interestbecause of the inherent sensitivity of optical image resto-ration algorithms to changes in the observations of thePSF.

Our restoration method has computational complexityof only O(ln2 log n), where n2 is the number of pixels inthe image and l is the number of segments used. More-over, parallelization of algorithm 2 seems a natural stepto take. The idea is that every image segment can beprocessed concurrently in l different nodes. With use ofthe master/slave paradigm, a master node determineswhen the inner preconditioned linear systems are solvedand when other tasks, such as H * wk21

(i) , are performed.By simulating a distributed data structure, where eachslave node i maintains local variables associated withsegment i, communication can be minimized to mostlysending and receiving instructions. The only time whenheavy communication occurs and synchronicity is re-quired is when the master node forms the solution of anouter preconditioned system. We have a parallelizedversion of algorithm 2, and it is implemented by using aparallel toolbox for MATLAB. It is not the purpose of thepresent paper to give an analysis of such a parallel algo-rithm. However, in our experiments, we note that thecomputing time of the parallelized version of SVR using3–5 segments was no more than that of MR-II with pre-conditioning applied to a single segment.

4. NUMERICAL TESTSIn this section we present numerical tests of three datasamples to illustrate the effectiveness of the SVR methodin comparison with the unpreconditioned MR-II andpreconditioned-based regularized MR-II methods inHanke and Nagy.23 The first data sample consists oftrue and degraded images of a simple known block object.The second data sample consists of true and degraded im-ages of simulated satellite data as they would be capturedby a ground-based telescope using an adaptive-optics sys-

tem. The third and last data sample includes actualground-based telescope images obtained at the U.S. AirForce Phillips Laboratory Starfire Optical Range by usingan adaptive-optics imaging procedure. Here, TV minimi-zation is used for the preprocessing step of denoising theguide star. For the SVR method, the degraded images ineach sample were partitioned into three segments by us-ing the segmentation scheme described in Section 3.

Example 1: Regular boundaries. This example con-sists of a simple 256 3 256 image (with 65,536 unknownpixel values) containing two blocks having different gray-scale intensities with a common boundary, as shown inFig. 4(a). To obtain a blurred image, we used theGaussian-type PSF shown in Fig. 4(b) (and described infurther detail in example 2) and convolved it with theoriginal image. The degraded image was then obtainedby adding 10% normally distributed random noise, withmean 0 and variance 1.0, yielding a degraded image hav-ing a SNR of

Fig. 5. Relative errors for block image data. ‘‘Prec’’ stands forpreconditioned.


SNR 5 10 log10~ iHf i2/ihi2! 5 20 dB.

This degraded image is shown in Fig. 4(c).The L-curve method was used to select the tolerance or

truncation parameters t i for the preconditioners Qtiasso-

ciated with each segment. This procedure is described inSubsection 2.B. Recall that the truncation parameter t iseparates the spectrum into signal and noise subspaces,

thereby enabling the space-varying preconditioner tocluster the singular values of H associated with the signalsubspace and to act like the identity on the noise sub-space. For the regularized preconditioned MR-II method(SVR with one segment), the truncation parameter thatwe obtained resulted in a signal subspace involving 350eigenvalues; that is, 350 eigenvalues were greater thanthe truncation parameter [following the definitions in

Fig. 6. Computed restorations of the block image.


Fig. 7. Simulated ground-based telescope image data: (a) true image, (b) PSF, (c) degraded image.

Eqs. (12) and (13)]. For the SVR method, the truncationparameters that we obtained for each of the three regionsresulted in signal subspaces involving 100 eigenvalues forthe region associated with the background, 350 eigenval-ues for the region associated with the border regions, and450 eigenvalues for the region containing most of the ob-ject (following the discussion of Subsection 3.B).

Because we have the true solution, for each method weare able to quantify solutions in terms of their relative er-rors. A plot of the relative error norms is shown in Fig. 5.As can be seen from the plot, the SVR method allows us toobtain smaller relative errors and, in that sense, bettersolutions. Moreover, SVR appears to stabilize the itera-tions, in that the relative errors are reduced quickly butthen remain at approximately the same level for many it-erations. For instance, comparing the relative errors atthe 15th iteration, we see that SVR still provides a goodrestoration, whereas restorations by the other methodsare corrupted with noise. This is a distinct advantage,since it is difficult (see, e.g., Hanke and Hansen11) to pre-dict the optimal stopping point in practice, and hence be-ing a few iterations further away from the minimum rela-

Fig. 8. Relative errors for simulated satellite image data.

tive error solution will not have much effect on thecomputed restoration. Thus robustness is incorporatedinto our SVR scheme, leading to more stability in the it-erations. Figure 6 shows the computed restorations foreach method, which further illustrates the stable behav-ior of SVR.

Example 2: Simulated ground-based telescope data.The image boundaries in example 1 are very regular, andhence the segmentation used to construct the SVR pre-conditioners is relatively simple to obtain. In the nextexample, consider a 256 3 256 image with irregularboundaries. These data were obtained from the U.S. AirForce Phillips Laboratory, Lasers and Imaging Director-ate, Kirtland Air Force Base, New Mexico. Specifically,the true object is an ocean reconnaissance satellite, whichis shown in Fig. 7(a). A computer simulation algorithmat Phillips Laboratory was used to produce an image ofthe satellite, shown in Fig. 7(c), as would be observedfrom a ground-based telescope by using adaptive-opticscompensation.2 The satellite was modeled as being 12 min length and in an orbit 500 km above the surface of theEarth. The simulated charge-coupled device for formingthe image was a 65,536-pixel square array. Charge-coupled device root-mean-square readout noise variancewas fixed at 15 mm per pixel to reflect a realistic state-of-the-art detector.

In actual field experiments, several hundred measure-ments are averaged to reduce the effects of noise. ThePSF and the degraded image used in these data, whichare shown in Fig. 7(b), were created to simulate thissituation.33,34

Again we use the L-curve method to select the toler-ance parameters for the preconditioners. When precon-ditioning on only one segment, following Eq. (12), this cor-responded to a signal subspace involving 1335eigenvalues. With SVR, and following Eq. (12), the Lcurve suggested using 600, 1335, and 1347 eigenvaluesfor the signal subspaces of the three respective regions.A plot of the relative errors is shown in Fig. 8. Observethat the SVR scheme does not obtain quite as small arelative error as that from the nonpreconditioned MR-IImethod. We attribute this observation to the low noiselevels in the data, which essentially reduces the need forregularization in this particular problem. However, theSVR algorithm exhibits a robust and stable behavior in


the sense that the relative errors decrease quickly and re-main at a low level for many iterations. Computed res-torations at various iterations are shown in Fig. 9.

Example 3: Ground-based telescope data. We con-sider here a real ground-based telescope image taken by aU.S. Air Force facility of a satellite circling the Earth anda corresponding PSF image of a nearby natural guidestar, Deneb. The guide star was chosen in the near fieldof view for the satellite, in order to approximate spatial

invariance of the PSF for the operator equation (1).Deneb is Arabic for tail, and this star is the tail of theCygnus constellation, a great swan flying southwarddown the Milky Way. It is a brilliant white star with vi-sual magnitude 1.4. Deneb and wing stars of the swanform an asterism called, for obvious reasons, the North-ern Cross. Figure 10 shows the observed degraded imageand an image of the guide star Deneb. We remark thatthe effects of atmospheric turbulence have been partially

Fig. 9. Computed restorations of the simulated satellite image.


compensated by an adaptive-optics system4 as the satel-lite and guide star were imaged through the telescope.The adaptive-optics system used was controlled by a pro-cedure similar to that developed by Ellerbroek et al.2,3

Figure 11(a) shows a surface plot of the observed PSFimage, and it is not difficult to see that the image containsa substantial amount of noise. A preprocessing step toreduce this noise is necessary to obtain a better restora-tion. In our experiments we attempted to restore the im-age both with and without denoising the PSF and foundthat significantly better restorations were obtained whendenoising. We tried various techniques for removingnoise from the PSF, such as local averaging, median fil-tering, and TV minimization. Best results were obtainedwith the TV method, as described in Subsection 2.A.Here, the sharp spike associated with the PSF is pre-served very nicely. Figure 11(b) shows a surface plot ofthe TV-denoised PSF. We mention that because the TVmethod requires solving a nonlinear minimization prob-lem, in general it can be expensive to implement. How-ever, the extent of the PSF is generally very small com-pared with the dimensions of the image, and so the cost ofdenoising the PSF is reasonably inexpensive comparedwith the cost of restoration.

It should be emphasized that these data are real andthat a true image of the object is not available. Therefore

Fig. 10. Real ground-based telescope data: (a) observed image,(b) observed PSF (Deneb).

we cannot use a plot of the relative errors to determine astopping criterion. Many methods have been proposedfor terminating the iterations, including the discrepancyprinciple, generalized cross validation, and the L-curvemethod.11 There are advantages and disadvantages toeach of these techniques, and we do not want to advocateany one method or go into any detailed discussion ofthem. However, a significant observation from our pre-vious tests is the stable behavior of the SVR iterates.This suggests that examining the norm of the differencein successive iterates may determine when the computedrestoration can no longer be improved significantly. Fig-ure 12 shows computed restorations for each of the im-ages at various iterations. Once again we observe thestability of the restorations using the SVR method.

5. CONCLUSIONSIn this paper we have presented a new segmentation-based preconditioning method, which we refer to as SVR,leading to space-varying regularized iterative restorationof optical images. Denoising methods, including totalvariation minimization, followed by segmentation-basedpreconditioning methods for minimum residual (MR) con-jugate gradient iterations, were investigated. Numericaltests were made on both simulated and practical atmo-spheric imaging problems. The SVR preconditioningmethod appears to have certain advantages:

• Even if a stopping criterion does not halt the algo-rithm precisely at the appropriate solution (either a fewiterations before or after), it appears that the SVRmethod will still provide a restoration that is close to op-timal. We have observed that the SVR method rapidlyreduces the relative error and then stabilizes at a levelclose to that of a near-optimal solution for several itera-tions. Examining the norms of the difference of succes-sive iterations, in conjunction with one of the standardstopping criteria, leads to a robust iterative method.

Fig. 11. Preprocessing of PSF: (a) observed PSF, (b) TV denoised PSF.


• As described in Section 3, the SVR preconditioningscheme can be easily implemented in parallel. In thiscase the total computational cost of our scheme is essen-tially the same as that for the fast preconditioned MR-IIrestoration method, which has computational complexityof only O(n2 log n), where n2 is the number of pixels inthe image.

• The appropriate segmentation procedure used, aswell as the number of segments chosen, is problem depen-

dent. More studies and comparisons are needed in thisregard. Analyzing the image in a well-chosen, piecewisesmooth form appears to yield better overall separation ofthe signal and noise subspaces for the purpose of choosingthe space-varying preconditioners. The resulting itera-tive restoration using SVR can effectively suppress noisecontamination of the iterates. Oversmoothing the imageis avoided, and singular image features such as sharpedges and oscillatory textures can be restored without re-

Fig. 12. Computed restorations of the real satellite image.


sorting to a computationally expensive nonlinear optimi-zation method such as total variation restoration.

ACKNOWLEDGMENTSThe authors thank R. Carreras, B. Ellerbroek, and C.Matson from the U.S. Air Force Phillips Laboratory, La-sers and Imaging Directorate, Kirtland Air Force Base,New Mexico, for supplying much of the simulated andreal data used in the numerical tests. We also thank R.Chan from the Chinese University of Hong Kong for sup-plying a section of the total variation code used to denoisethe guide star Deneb.

Through the use of the Maui High Performance Com-puting Center, this research was sponsored in part by thePhillips Laboratory, U.S. Air Force Materiel Command.Individual authors were supported in part as follows: J.G. Nagy by a National Science Foundation PostdoctoralResearch Fellowship; V. P. Pauca by the U.S. Air ForceOffice of Scientific Research; R. J. Plemmons by the U.S.Air Force Office of Scientific Research and by the NationalScience Foundation; T. C. Torgersen by the National Sci-ence Foundation.

REFERENCES1. J. Brown, ‘‘Overview of topical issues on inverse problems

in astronomy,’’ Inverse Probl. 11, 635–638 (1995).2. B. Ellerbroek, C. Van Loan, N. Pitsianis, and R. Plemmons,

‘‘Optimizing closed-loop adaptive-optics performance usingmultiple control bandwidths,’’ J. Opt. Soc. Am. A 11, 2871–2886 (1994).

3. B. Ellerbroek, C. Van Loan, N. Pitsianis, and R. Plemmons,‘‘Jacobi-like method for a problem arising in adaptive-optics,’’ Tech. Rep. (U.S. Air Force Phillips Laboratory, La-sers and Imaging Directorate, Kirtland Air Force Base,N.M., 1996); submitted to SIAM J. Opt.

4. J. W. Hardy, ‘‘Adaptive-optics,’’ Sci. Am. 270 (No. 6), 60–65(1994).

5. R. Lagendijk and J. Biedmond, Iterative Identification andRestoration of Images (Kluwer, Boston, 1991).

6. J. Nagy, R. Plemmons, and T. Torgersen, ‘‘Fast restorationof atmospherically blurred images,’’ in Advanced SignalProcessing: Algorithms, Architectures, and Implementa-tions IV, F. T. Luk, ed., Proc. SPIE 2027, 542–553 (1993).

7. J. Nagy, R. Plemmons, and T. Torgersen, ‘‘Iterative imagerestoration using approximate inverse preconditioning,’’IEEE Trans. Image Process. 5, 1151–1162 (1996).

8. J. Biedmond, R. Lagendijk, and R. Mersereau, ‘‘Iterativemethods for image deblurring,’’ Proc. IEEE 78, 856–883(1990).

9. R. Chan, J. Nagy, and R. Plemmons, ‘‘FFT-based precondi-tioners for Toeplitz-block least squares problems,’’ SIAM(Soc. Ind. Appl. Math.) J. Numer. Anal. 30, 1740–1768(1993).

10. N. Clinthorne, T.-S. Pan, P.-C. Chiao, W. L. Rogers, and J.A. Stamos, ‘‘Preconditioning methods for improved conver-gence rates in iterative reconstructions,’’ IEEE Trans. Med.Imaging 12, 78–83 (1993).

11. M. Hanke and P. Hansen, ‘‘Regularization methods forlarge-scale problems,’’ Surv. Math. Ind. 3 253–315 (1993).

12. S. Reaves, ‘‘Optimal space-varying regularization in itera-tive image restoration,’’ IEEE Trans. Image Process. 3,319–324 (1994).

13. L. Alvarez and J. Morel, ‘‘Formalization and computationalaspects of image analysis,’’ Acta Numer. 3, 1–59 (1994).

14. S. Osher and L. Rudin, ‘‘Feature-oriented image enhance-ment using shock filters,’’ SIAM (Soc. Ind. Appl. Math.) J.Numer. Anal. 27, 919–940 (1990).

15. R. Acar and C. Vogel, ‘‘Analysis of bounded variation pen-alty methods,’’ Inverse Probl. 10, 1217–1229 (1994).

16. D. Dobson and F. Santosa, ‘‘Recovery of blocky images fromnoisy and blurred data,’’ SIAM (Soc. Ind. Appl. Math.) J.Appl. Math. 56, 1181–1198 (1996).

17. L. Rudin and S. Osher, ‘‘Total variation based image resto-ration with free local constraints,’’ in Proceedings of theIEEE International Conference on Image Processing, II(IEEE, New York, 1994), pp. 31–35.

18. L. Rudin, S. Osher, and E. Fatemi, ‘‘Nonlinear total varia-tion based noise removal algorithms,’’ Physica D 60, 259–268 (1992).

19. C. Vogel and M. Oman, ‘‘Iterative methods for total varia-tion denoising,’’ SIAM J. Sci. Comput. 17, 227–238 (1996).

20. A. S. Nemirovskii, ‘‘The regularizing properties of the ad-joint gradient method in ill-posed problems,’’ USSR Com-put. Math. Math. Phys. 26, 7–16 (1986).

21. O. Axelsson, Iterative Solution Methods (Cambridge U.Press, Cambridge, 1994).

22. M. Hanke, J. Nagy, and R. Plemmons, ‘‘Preconditioned it-erative regularization of ill-posed problems,’’ in NumericalLinear Algebra and Scientific Computing, L. Reichel, A.Ruttan, and R. Varga, eds. (Walter de Gruyter, Berlin,1993), pp. 141–163.

23. M. Hanke and J. Nagy, ‘‘Restoration of atmosphericallyblurred images by symmetric indefinite conjugate gradienttechniques,’’ Inverse Probl. 12, 157–173 (1996).

24. M. Hanke, Conjugate Gradient Type Methods for Ill-posedProblems, Pitman Research Notes in Mathematics (Long-man Scientific and Technical, Harlow, Essex, UK, 1995).

25. M. Paige and M. A. Saunders, ‘‘Solution of sparse indefinitesystems of linear equations,’’ SIAM (Soc. Ind. Appl. Math.)J. Numer. Anal. 12, 617–629 (1975).

26. T. F. Chan and J. A. Olkin, ‘‘Preconditioners for Toeplitz-block systems,’’ Numer. Algor. 6, 89–101 (1993).

27. P. J. Davis, Circulant Matrices (Wiley, New York, 1979).28. P. Hansen and D. P. O’Leary, ‘‘The use of the L-curve in the

regularization of discrete ill-posed problems,’’ SIAM J. Sci.Comp. 14, 1487–1503 (1993).

29. C. Avila, ‘‘An adaptive regularized method for deconvolu-tion of signals with edges by convex projections,’’ IEEETrans. Signal Process. 42, 1849–1851 (1994).

30. S. Choy, Y. K. Chan, and W. C. Siu, ‘‘A new adaptive itera-tive image restoration algorithm,’’ in Proceedings of theIEEE International Conference on Image Processing, II(IEEE, Piscataway, N.J., 1994), pp. 670–674.

31. J. Nagy and D. P. O’Leary, ‘‘Restoring images degraded byspace variant blur,’’ submitted to SIAM J. Sci. Comput.

32. K. Castleman, Digital Image Processing (Prentice-Hall,Englewood Cliffs, N.J., 1996).

33. R. Carreras, U.S. Air Force Phillips Laboratory, Lasers andImaging Directorate, Kirtland Air Force Base, N.M. 87117(personal communication, 1995).

34. C. Matson and M. Roggemann, ‘‘Noise reduction inadaptive-optics imagery with the use of support con-straints,’’ Appl. Opt. 34, 767–780 (1995).

space-varying restoration of optical images

Documents