ieee transaction on image processing

8/9/2019 IEEE TRANSACTION ON IMAGE PROCESSING

1/5

1 I9E E E T R A N SA C T I O N S O N I M A G E PROCESSING. V O L . I . NO. I. J A N U A R Y 1992part o f the GM L algori thm is the convolu t ion operat ion and , there-fore, can be processed by me ans of the fast Fourier transform (FF T)technique [4], [6] .

A com mon perfor mance m etric for the motion analysis is themeasure of the match which may be defined as the difference be-tween the present frame and the displaced previous frame. The nor-malized mean squared error (NMSE) of this quantity is defined asfo l lows:

1N M SE =- C [ ~ ( i D ( i ) )- s( i - 8,- (i))] (17)N F Iwhere n = 0, 1 , 2 , . . ,8-,i) = 0, NF is a normal iz ing factor(to normalize the NMSE to unity in the first i teration), and i =( i, j ) T where i, j = 1, 6 4 .

Fig . 2 i l lustrates a noise-free scen ario for which NMS E is cal-cu la ted for + and the multiples of +. As seen in this figure, one canenhance the adaptation speed of the algorithm by increasing +within a certain limit. Fig. 3 highlights the effect of the variationof in a high noise case (noise variance is 2000). This figure ver-ifies that the algorithm is m ore sensitive to the converg ence param-eter in noisy situations. Finally, Fig. 4 exhibits the effect of dif-ferent noise levels on the convergence of the GML algorithm andverifies the stabili ty under various noise conditions. In this exper-iment, the convergence parameter is set to the value of lo + and thenoise variances are 0, 300, and 2000.

I V . S U M M A R YND C O N C L U S I O NThis paper has estab l i shed and implemented a formulat ion for

presetting the convergence parameter of the GML algorithm for theframe-to-frame motion estimation.

It should be emphasized tha t the analysis presented here is basedon the assumptions that a) the elements of the motion vector areuncorre la ted , b) the convergence parameters are equal , c ) the mo-tion and its corresponding estimates are relatively small such thatthe first-order approximations are valid, and d) convergence isstudied for the a v e r a g e of the estimation error and in the absenceof noise; clearly, these assumptions may be severe in some cases.In these events, one should adjust the theoretical range of valuessuggested by (13) and (14) for proper functioning of the GML al-gorithm (3).

A C K N O W L E D G M E N TThe authors would like to thank the anonymous reviewers for

their constructive and helpful comments and suggestions. Thanksare a lso ex tended to S. Shal taf , J . Lipp, and Professors D. B.Brumm and C. R. Givens for thei r technical support during thecourse of this investigation.

I11121

r31

141

R E F E R E N C E SG . Musmann er a l . , Advances in picture coding, P roc . IEEE, vol.73 , pp . 523-548, Apr. 1985.N. M. Namazi and J . A. Stuller, A new approach to signal registra-tion with the emphasis on variable time delay estimation, IEEE Trans.Acoust. , Speech Signal Processing, vol. ASSP-35. pp. 1649- 1660,Dec. 1987.N. M. Namazi and C. H . Lee, Nonuniform image motion estimationfrom noisy data, IEEE Trans. Ac ous t. , Speech Signal Proce ssing,vol. 38, pp. 364-366, Feb. 1990.T. J . Beukema and N. M . Namazi, FFT implementation of a frame-

to-frame motion estimator, Int. J . Robotics Automation, vol. 5 , no.4, Dec. 1990.151 G . Strang, Linear Algebra and It s Applications. New York: Aca-demic, p. 202, 1976.161 N. M. Namazi and D. W . Foxall, Use of the generalized maximumlikelihood algorithm to estimation of the Markovian modelled imagemotion, J . O p t . E n g . , vol. 30, no. 10, pp. 1486-1489, Oct. 1991.

Automatic Assessment of Constraint Sets in ImageRestoration

Stanley J. Reeves and Russell M. Mersereau

Abstract-Constraints provide an importa nt mean s of incorpo ratinga priori information into the image restoration process. However, muchof the information available for constructing constraints is of a tenta-tive nature. If the validity of this tentative information can h e assessedbefore it is incorporated into the solution, helpful constraints can heretained while harm ful ones can h e discarded.Cross validation is introduced as a technique for assessing the valid-ity of such constraint sets. Because the full c ross validation procedureis computationally burdensom e, a modification is suggested that allowsa more feasible implementation without substantially sacrificing theperformance of the full procedure. Experimental results demonstratethe excellent performance of both the full an d modified procedures.

I . I N T R O D U C T I O NMany image deg radation problems can be modeled as a l inear

blur in the presence of additive noise. This degradation process isrepresented in terms of the equation

g = D f + n (1 )where D s the operator that describes the linear degradation pro-cess , f an d g are the lexicographically ordered original and de-graded images, and n is independently distributed random noiseuncorrelated with the imagef. The goal is to recover an estimateo f fg i v en o n l y g an d D . Howev er, the degradat ion due to b lur andnoise generally makes estimating the original image difficult . Theinverse of the blur operator may be nonunique, and the noise tendsto be amplified unacceptably.

The degradation introduced by blur and noise can be viewed asa loss of information about the original scene. The restorationproblem is then defined as an attempt to supply lost informationsuch that the original scene is recovered as closely as possible.Constrained image restoration has been widely recognized as a vi-able technique for incorporating a p r i o r i informa tion into the imagerestoration process [I], [2]. Promising results have recently beenreported in [3] using a bounded local variance constraint. Otherconstraints are also possible. A number of physically meaningfulconst ra ints are g iven in [ I] .

Constraints on the restoration can often be used to supply someof the missing information. However, the assumptions used to con-

Manuscript received March 28 , 1990; revised June 2 5 , 1991. This workwas supported in part by the Joint Services Electronics Program under Con-tract DAAL-03-90-C-0004.S . J . Reeves is with the Department of Electrical Engineering, AuburnUniversity, Auburn, A L 36849-5201.R. M. Mersereau is with the School of Electrical Engineering, GeorgiaInstitute of Technology, Atlanta, GA 30332-0250.IEEE Log Number 9104336.

Authorized licensed use limited to: Proof and Experimental Establishment. Downloaded on December 26, 2009 at 11:01 from IEEE Xplore. Restrictions apply.


2/5

120 IEEE T R A N SA C T I O N S ON I M A G E PROCESSING. VOL. I . N O . I . J A N U A R Y 1992struct a constraint set are often tentative. Thus a given const ra in tmay or may not improve the restoration result . If the validity of aconstraint can be assessed before it is imposed on the solution,constraints representing helpful information can be retained whilethose represent ing harmful informat ion can be d iscarded .

Determining the validity of constraints in image restoration byvisual inspection often proves cumbersome and sometimes unreli-ab le . Therefore , a m ethod for assessing const ra in ts au tomat ical lyand objectively is desirable. This correspondence presents a tech-nique that yields objective measures for assessing the validity ofconstraint sets for constrained image restoration. The method isbased on the principle of cross validation. Because the full crossvalidation procedure is computationally burdensome, a simpler ap-proach is suggested that allows a more feasible implementationwithout substantially sacrificing the performance of the full pro-cedure . Experimenta l resu l t s demonst ra te the excel len t perform-ance of both the full and simplified procedures in assessing con-straints for image restoration.

1 1 . C O N STR A IN EDM A G E E S T O R A T IO NIn this paper, constrained image restorarion refers to image res-

toration tec hniques that constr ain the restoration result to l ie withina specified set. Such constraints are intended to restrict the admis-sible class of restorations to those that are considere d feasible. T hisapproach allows the incorporation of a pr ior i information about therestoration.

Constraints that can be expressed as projections onto convex setsare extremely useful. If a signal x lies within a convex constraintse t C , i t will be unchange d by the projection oper ator onto that set;that is, a signal x satisfies the constraint if and only if

x = Px (2 )where P is the projection operator for the constraint set C.

Achieving a constrained image restoration often requires i tera-tive restoration techniques. The iterates can be defined as fo l lows:

(3)(4 )

where P is a composite constraint set projection operator of theform PIP 2 . . . P,. (PA epresents the pro ject ion operator fo r thekth convex set C, in which the signal is restricted to lie.) Givenproper choices of the re laxat ion parameter p,, and the correctionterm pn , he image will converge to the constrained least-squaressolution [2]. If 0,= 0, the preceding iteration reduces to the methodof pro ject ion onto convex se ts (POC S). T he so lu t ion i s guaranteedto lie in the intersection of the constraint sets as long as the inter-section is not the empty set. Thus constrained restoration isstraightforward using iterative techniques.

111. T H E M E T H O D F C R O S SV A L I D A T I O NThe image restoration field clearly needs a method for testing

assumptions imposed on the restoration process. If such a test wereavailable, then assumptions could be either validated or re jec ted ;in this way the restoration could proceed using only those assum p-tions that are considered valid. Cross validation provides such amethod.

Cross validation has long been recognized as a useful techniquein statistical data analysis [4]. It has also been employed to estimatethe proper degree and type of smoothing in image restoration 1.5-171. Motivated by the successful application of cross validation to

other image restoration problems [6]-[8], we have extended it toaddress the const ra in t assessment problem.

The central idea behind cross validation is an old one-the dataset is divided into an estimation set and a validation set. A portionof the data is used to obtain a model or estimate based on a partic-ular assumption. The other portion of the data is then used to val-idate the performance of the model or estimate and thus the as-sumption. In this way, various candidate assumptions can be testedso that the b est choice can be selected on the basis of perform-ance with the validation data. The inevitable dilemma faced by thisdivision is that i t is desirable to use as much of the data as possiblefor obta in ing an est imate , bu t it is also important to test the modelon data that was excluded from the model building process.

Cross validation is an attractive way of dealing with this di-lemma. Instead of using one portion of the data exclusively forestimation and the other for validation, cross validation allows allthe data to be used for both purposes. The technique works as fol-lows. The data set is divided into M sets (M > 1). The assumpt ionbeing tested is imposed on al l the sets but one, and a validationerror measure is compu ted for the left out set. The assum ption isthen imposed on the data again, for a total of M t imes, alternatelyleaving out each set in the estimation procedure and using that setto obtain a validation error measure. In this way all the data can beused for both estimation and validation.A . Definition of Error Terms

Because constraints are simply a particular type of assumption,cross validation provides a means of checking the validity of con-straints. Let the observed image g be divided into M distinct setsof points GI such that

MU G , = Gh = I

( 5 )where G represents all the points in the observed image. For no-tational consistency Nh will represent the number of points in theset G,. Le t f represent the f E C that minimizes the modelingerror:

where P is the projection operator that corresponds to the constraintse t C and the terms g, and [D f][ refer to the ith entries of the vectorsg an d D f. The term ~ ~ c i L f ~ ~ *s a stabilizing term that invokes Tik-honov-Mil ler regularizat ion [2]. The operator L is usually chosento be the discrete Laplacian operator, as i t was for the experimentshere . T he val idat ion error i s

(7 )I MV ( P ) = - C C ( g , - [DfkI , ) .Nh=l r e GThe choice of the sets Gk over which to compute the predictionerror is not extremely crit ical. Th e particular pixels chosen for eachset will affect the predicted values of the other pixels in the set;however, comput ing the error over a large number of points shouldaverage out the effect of the interactions between the pixel values.Experiments showed that for M < 4, the cross val idat ion error i srather sensitive to the method used to compute the restoration. Wechose the validation points in the sets {G I randomly rather thanby scan line or some other regular pattern. Results tended to bemore reliable using a random choice of points.

The validation error can be used in one of two ways. First , theerror can be computed with and without a particular constraint. Ifthe constrained validation error is lower, then the constraint is

Authorized licensed use limited to: Proof and Experimental Establ ishment. Downloaded on December 26, 2009 at 11:01 from IEEE Xplore. Restrictions apply.


3/5

I E E E T R A N S A C T I O N S ON I M A C k P K O C E S S I N G . V O I . I . N O I . . l i \ N U A K Y IW ?

Fig. I . Constrained restoration examp le. (a ) Original cameranian image.( b ) 9 X 1 uniform linear motion blur with 30-dB BSNR. ( c ) Restorationobtained with [O . 2551 intensity constraint. (d )Constrained restoration withT = 4.8-cross validation and minimum M SE choice.

judged to improve the restoration. Otherwise. the constraint is con-sidered erroneous. S econd, thc validation error for a number ofcompeting constraints can be computed and the constraint with thelowest error chosen as the best. In either case , the cross vali-dation method supplies a valuable technique for constrained imagerestoration.B. Simplijied Criterion

Because most images have an ex t remely largc number of p ixels .we investigated the possibili ty of approximating the cross valida-tion error by computing the error over only a single deleted set ofpixels rather than all M sets. In this way only a fraction of th eobservations yields a prediction error, but a single restoration suf-fices to compute the error function. With f; defined as the mini-mizer of (6) for k = I , th e error function is

image, the modified error measure will be a good approximation tothe same nieasure compu ted ove r all the sets. The crit ical ditferencebetween the ful l and modified measures is the variance of the es-t imates . Experiments showed that for N , = ( 1 / 1 0 ) N , hc image issufficiently sampled t o yield a good approximation of the nieasurein (7 ) . This approach proved to be an effect ive approximat ion tofull cross validation.

A modified conjugate gradients algorithm was used to minimize(6) for a part icu lar choice of the projection operator P . Th e co n -jugate gradien ts a lgori thm converges faster than the s teepest de-scent method and is not very sensitive to the nonlinearities intro-duced by the projection operator. The same algorithm was used toobtain the restoration result for fixed P .

IV . E X P ~ K I M E N T SWe blurred the cameraman image (Fig . l (a )) wi th a 9 X 1 uni-

form horizontal motion blur and added noise at levels of 20-, 30-,and 40-dB b lurred s ignal -to-noise ra t io (BSNR ), where

(9)blurred image powernoise powerwhere the subscrip t DD is for data division. This simplificationof cross validation is equivalent to the holdout method. However,because this can also be viewed as a sampled cross val idat ionmeasure , we cont inue to refer to it as a simplified cross validationprocedure . The choice of the set G I is not critical. As long as thepoints in G I are well-distributed ov er the various features in the

BSNR = 10 lo gFig . l (b) shows the b lurred and noisy image for the 30-dB case . Adiscrete Laplacian was used as the regularization operator with a= 0 . 4 7 4 , 0.15, and 0 .0474, respect ively , for th e 20-, 30-, and 40-

Authorized licensed use limited to: Proof and Experimental Establ ishment. Downloaded on December 26, 2009 at 11:01 from IEEE Xplore. Restrictions apply.


4/5

122 IEEF

-

--

-

-

TRANSACTIONS ON IMAGE PROCESSING, V O L . I . N O . I. JA N U A R Y 1992

233

218

204

189

174

- 560

- 520

- 480

8 - , -

220 440

4005 15 25 300

I6 3

T

Fig. 2 . MS E (dashed), cross validation (solid), and D D errors (dotted)versus constraint threshold Tf or the 20-dB case. (The scale for MS E is onthe right side of the plot.)

dB cases . Both the fu l l cross val idat ion error and the data DD errorwere computed for each const ra in t se t . The D D error was computedusing each of the pixel sets { G k } .W e u s ed M = 10, with eachpixel assigned to one of the ten sets randomly, with equal proba-b i li ty for each se t . T he to ta l number of p ixels for th is case was N= 6 5 5 3 6 .

We est imated a local variance measure for the cameraman im ageusing the blurred image data. Local variance was defined as

r + P , + f J

(10)where the subscript ( i , j ) enotes a 2-D representa tion of the imagesused and m, i, j ) s the local mean given by

r + P i+Qc c g ( k , I ) . ( 1 1 )1( 2 P + 1) (2 Q + 1) L=,-,, / = / - a, ( ; , j ) =For th is experiment , we se t P = Q = 2 . A class of constraint setswas developed from the local variance measure such that

for various values of T. This constraint has the effect of smoothingthe restoration in regions where the local variance is low. Crossvalidation was used to determine the optimal choice of the thresh-old T.

Error curves for the three cases are shown in Figs . 2 -4 . Thecross validation error (solid l ine) followed the shape of the MSEcurve (dashed l ine) qui te c losely in ea ch case . T he dot ted l ines arethe DD curves f or each of the ten se ts . The D D curves a lso fo l -lowed the general shape of the MS E curve, a l though not a lways asclosely as the cross validation curve. In a few cases , the minimizerof a DD curve did not coincide with the minimizer of the crossval idat ion curve , bu t the DD minimizer was a lways qui te c lose .Both cross val idat ion and the majori ty of the D D m easures choseT in harmony wi th the minimum MSE choice for 20 an d 30 d B .F o r 40 dB, the const ra in t provided only marginal improvement ,and the cross val idat ion and DD minimizers were s l igh t ly sh i f tedfrom the minimum MSE minimizer. The image obta ined wi th the

30 ..................... ~ .......t

I1 6 01 2 3 4 5 6 7T

Fig. 3. M SE (dashed), cross validation (solid). and D D errors (dotted)versus constraint threshold T for the 30-dB cas e. (The scale fo r M SE is onthe right side of the plot.)

1 I I I , I I , 1 6 13O 0.5 1 1.5 2 2.5 3 3.5 4 4.5 S

T

Fig. 4. MS E (dashed), cross validation (s olid), and D D errors (dotted)versus constraint threshold Tf or the 40-dB case. (The scale for MS E is onthe right side of the plot.)

opt imal T(= 4.8) for the 30 dB case is shown in Fig . l (d) . Fig . l (c)shows the result of the image restored with only the [ 0 , 25S] nten-sity constraint (T = 0). Further resu l t s for th is technique can befound in [9].

V . C O N C L U S I O NWe have presented cross val idat ion as a m ethod for assessing the

validity of constraint sets in the context of constrained image res-toration with promising results. Because the full cross validationprocedure i s computat ional ly burdensome, the holdout method wasimplemented as a feasible alternative without sacrificing substan-tially the performance of the full procedure. The large amount ofdata in a typical image allows the use of only a representative setfor estimating the validation error without substantially affectingperformance. T his approach y ie lds reasonable resu l t s , a l though notas reliable as the full cross validation measur e.

Given that the model expressed by (1 ) is valid, cross validationrequires no o ther assumpt ions about the data to perform sat i sfac-torily. Knowledge of the amount of noise or i ts distribution is alsounnecessary . As long as the degradation model is correct, cro ssvalidation should provide useful information about the validity ofconstraints in any signal restoration problem.

Authorized licensed use limited to: Proof and Experimental Establishment. Downloaded on December 26, 2009 at 11:01 from IEEE Xplore. Restrictions apply.


5/5

I E E E T R A N SA C T I O N S ON I M A G E PROCESSING. VOL. I . N O I . J A N L I A R Y 1992 12 3

REF ERE NCE S est imates of image mot ion to restore the degraded image in an[ I ] D . C. Youla and H. Webb, Image restoration by the method ofcon-vex projections: Part I -theory, lEEE Truns. Mecl. Imaging. vol.121 A. K. Katsaggelos, Con strain ed itera tive image restoration algo -rithms, Ph.D. dissertation, Georgia Instit. of Technol., 1985.[3] M. I . Sezan and A . M . Tekalp, Adaptive image restoration with ar -tifact suppression using the theory of convex projections, lEEE Trans.Acous t . . Speech, SiRnal P rocess ing , vol. 38. pp. 181-185. Jan. 1990.[4] M . Stone, Cross-validatory choice an d assessment of statistical pre-diction, J . Royal Statist ical Soc. B , vol. 36. pp. 1 1 1-147. 1974.151 J. W . Kay, On the choice of regularisation parameter in image res-toration, in Pattern Recognition, 4th lnr. Conf P r o c . . 1988, pp. 587-596.161 S. J . Reeves and R. M . Mersereau , Regularization parameter esti-mation for iterative image restoration in a weighted Hilbert space, inP roc . 1990 IEEE ln t . Con$ on Acoustics, Speech, and Signal Pro-cess ing , 1990, pp. 1885-1888.(71 -. Optimal estimation of the regularization parameter and stabiliz-ing functional for regularized image restoration, O p t . E n g . , vol. 29,

DD . 446-454. Mav 1990.

M I - I , pp. 81-94, Oct. 1982.adapt ive manner.

Previous work on space-variant image restoration is relativelysparse . M ethods include coord inate t ransformat ions [ I ] , sectionalprocessing [ 2 ] , and Kalman fil tering [3]. The approach using co-ordinate transformations is l imited to types of motion such thatt ransformat ions can b e derived to chan ge the space varian t probleminto a space invariant one. T his has been done fo r rotational motionblurs and some types of geome trically induced motion blurs, suchas produced in satelli te photography. Sectional methods and Kal-man fil tering can be used to treat a more general class of blurs, butare l imited by the availabili ty of goo d estimates for the local deg-radation of the image. In this paper, the problem of estimation ofthe local mot ion parameters for sect ional methods i s done usingadjacent frames from a sequent ia l data se t .

11. I M A G EF O R M A T I O N ODE L..181 -. Identification of image blur parameters by the method of gen-eralized cross-validation, in P roc . 1990 ln t . Symp. on Circuits andSvstems. 1990. DD. 223-226. The images of i n t e r e s t ar e taken from a sequential set of d a t a ,Each of the recorded images is assumed to be obtained from the..191 S. J . Reeves, A cross-validation approach to image restoration andblur identification, P h.D. dissertation. Georgia Instit. of Technol.,1990. g = H f + ncomm on l inear model

Identification and Restoration of Spatially VariantMotion Blurs in Sequential ImagesH . J . Trussel l and S . Fogel

Abstract-Sequential imaging cameras are designed to record objectsin motion. When the speed of the objects exceeds the temporal reso-lution of the shutter, the image is blurred. Because objects in a sceneare often moving in different directions at different speeds, the degra-dation of a recorded image may he characterized by a space-variantpoint spread function (PSF). Th e sequential nature of such images canhe used to determine the relative motion of various parts of the image.This information can he used to estimate the space-variant PSF. A

where f is an M x 1 vector representing a frame of the originalsignal sequence, g is N x 1 representing the recorded information,n is N X 1 representing signal independent noise, and H is M XN representing the motion blurring.

For the case of space-invariant motion, the matrix H is Toeplitz,in the one-dime nsional case, or block Toeplitz in the two-dimen-sional case [4]. The computat ion i s carried out by making a c i r-cu lan t approximat ion to the Toepl i tz form and apply ing the DF Tto diagonalize the matrices. In the case of space-variant blurs, thematrix is not Toeplitz and fast restoration methods are not avail-able. However, using the sectional approach, i t is assumed that theimage can be processed in a local area where the blur can be treatedas spatially invariant.

Fo r this application, the two-dimensional problem differs fromthe one-dimensional problem only in the complexity of the index-ing . The fundamental mathemat ical problem is the same. Thus theone-dimensional case will be considered because of the simplicitymodification of the Landweher iteration is developed to utilize thespace-variant PSF to produce an estimate of the original image. of notation,The examples to demonstrate the effectiveness of th emethod are two d imensional .

I . IN TR O D U C TIO NSequent ia l imaging cameras are designed to record objects in

motion. T he objects may be moving in different directions at dif-ferent speeds. Often the speed of the objects exceeds the temporalreso lu t ion of the shut ter and the image i s b lurred . W hen such im-ages are analyzed on a frame-by-frame basis , i t i s desi rab le to re-duce the effects of such b lurring by image restora t ion . However,virtually all image restoration methods are designed to be imple-mented globally. This paper introduces a method of util izing local

Manuscript received May 7 , 1991; revised August 5 , 1991.H. J . Trussell was with Image Information Systems, Eastman KodakCompany, Rochester, NY. He is now with the Department of Electricaland Computer Engineering, North Carolina State University, Raleigh. NCS. Fogel is with Image Informations S ystems, Eastman Kodak Company.IEEE Log Number 9104328.

27695-79 1 1.Rochester, NY 14650-1816.

The sect ional approach can be described by considering a s inglesectionf,, and its adjacent neighbors. N otationally, this is writtena s

(2 )It is assumed that point spread function (PSF) of the center sectioncan be well-approxim ated by a space-invariant system ov er i ts localarea. This implies that the banded submatrix H I , ] s Toepl itz . Th eextent of the impulse response or PS F is assumed to be smal l wi threspect to the s ize of the submatrices . T his means that the subma-t r ices in the comer , H, + I - an d HI - I + ,, will be zero . T he off-diagonal matrices will have nonzero elements only in their comernear the diagonal. The restoration method presented in the nextsection can consider the effects of the adjacent sections.

A h i d li d li i d P f d E i l E bli h D l d d D b f IEEE X l R i i l

ieee transaction on image processing

Documents