Systems and Computers in Japan, Vol. 27, No. 14, 1996 Translated from Denshi Joho T h ~ ~ h i n Gakkai Ronbunshi, Vol. J78-D-11, No. 12, December 1995. pp. 1821-1830

Restoration of Space-Variant Blurred Image Using a Wavelet Transform

Shoichi Hashimoto* and Hideo Saito

Faculty of Science and Technology, Keio University, Yokohama, Japan 223

SUMMARY

This paper proposes a method to reconstruct a space-variant blurred image by using the wavelet trans- form, which is an orthogonal transform producing space-variant frequency data. A blurred image is di- vided into multiple resolutions by using a wavelet transform. As each resolution depends on the position of the image, this is corrected by applying a coefficient determined by a PSF parameter (the half-maximum width) so that the position-dependent frequency data are restored. The PSF in the blurred image is esti- mated to realize blind deconvolution. The method has been tested by a computer simulation and by a CCD camera using real images. The results show that the PSF in the blurred images can be estimated accurately when the images contain little noise. When the reso- lution of each blurred image is corrected by the esti- mated PSF parameter, it has been confirmed that the restoration was carried out depending on the degree of blurring.

Key words: Image restoration; wavelet trans- form; space variance; multiresolution representation; point spread function.

1. Introduction

With the development of photographic tech- nology, it is not possible to produce a high-quality

*Currently "IT

76

photograph, even for a simple snapshot. However, the problem of blurred images has not been solved, and various methods to restore such images have been proposed [l-31. However, most of those methods have been designed to restore a uniformly blurred image. But most blurred images are only partially blurred, for example, a poor focus for a main subject with a good focus for its background. No practical solution for this type of blurring has been proposed.

Generally, a model of a blurred image is given by

where f(x, y) is the original image, g(x, y) is its degraded image, h(x, y, s, t ) is a PSF, and n(x, y) is noise.

If the PSF in Eq. (1) is independent of a position in an image, it is possible to restore the degraded frequency data by using a simple division, since the blurred image is the product of the original image and the PSF in a Fourier space. If the blurring is dependent on position in the image, it is necessary to correct the frequency data depending on the position.

To realize this, both the data of position and frequency must be treated simultaneously. Therefore, a Fourier transform in which the position data are not easily obtained is not suitable.

Recently, the wavelet transform which can anal-

1SSN0882-1666/96/0014-0076 8 1997 Scripta Technica, Inc.

yze both position and frequency data at the same time has been popular, and its applications have been stud- ied in various fields [8-lo].

The wavelet transform is based on an arbitrary solitary wave, and this converges to zero at a finite distance, is localized on the time axis, and gives spa- tially localized frequency data. However, the Fourier transform is an orthogonal transform based on an infi- nite sinusoidal wave. Therefore, it is possible to apply the wavelet transform to a restoration of a blurred image.

This paper proposes a method of restoring a space-variant blurred image by using the wavelet trans- form. Also proposed is a method of estimating the area of a partially blurred region and its PSF in an image. For example, this method can be applied to a case where either a subject or its background alone is focused incorrectly. Then, an attempt is made to real- ize blind deconvolution [4,7], which restores a blurred image by using the given image and its preknown infor- mation alone.

2. Method of Restoring an Image Using Wavelet Transform

2.1. Wavelet transform

The wavelet transform is a method of time-fre- quency analysis using a solitary wave which is situated locally in a time region and a frequency region [ 11,121. This is represented by

where q() is the fundamental wavelet function consist- ing of a scale parameter u and a shift parameter b. The frequency band is controlled proportionally by u, and the wavelet function can be moved by b so that the change of a signal at an arbitrary position can be de- tected. Therefore, an arbitrary datum in a time series at an arbitrary time and arbitrary frequency band can be obtained by the wavelet transform.

Generally, it is not easy to obtain an orthogonal transform which satisfies the fundamental wavelet function. However, Mallat [13, 141 has related the wavelet to multiple resolution analysis, and has shown that the wavelet can be realized by recurrently applying a two-section filter (high and low bands) to the low- frequency side. Figure 1 shows a filter bank of 2D

Fig. 1. Filter bank of 2D wavelet transform.

Fig. 2. Multiresolution representation and Fourier space of 2D image.

transform. Each resolution decomposed by the wavelet transform is represented by a multiresolution as shown in Fig. 2. The frequency data of each resolution shown in Fig. 2 almost corresponds to the same pattern. Let us represent the resolution Wr for signal f by FZ where n = hh, hl, lh, and i = -1, -2, -3, ... .

It is considered that a space-variant blurred image has a different degree of blurring depending on the position in the image, even if the wavelet transform is applied, since the degradation of the frequency data also depends on the position. Therefore, if this state of degrading is corrected, the frequency data are also corrected.

2.2. PSF model

Let us make a model of a blurred image. In gen- eral, the state of blurring of an image can be repre- sented by the impulse response of a degraded system of a point light source. The form of the PSF function is determined uniquely by the cause of the blurring of the image. For example, the function is cylindrical for

77

I

Fig. 3. Method for decision of recovering value of each resolution.

an image out of focus, and is a line shape for a camera subject to vibration [l]. This paper treats blurred images caused by being out of focus which is consid- ered to be the most common cause of the degrading. The function of the PSF due to an out-of-focus image is cylindrical when the lens has a finite aperture with no aberration. However, a degrading caused by a prac- tical lens with an infinite aperture aberration and light attenuation, is represented by a Gaussian PSF as

where u2 is a variance, and u a r is a half-maximum width.

23. Space-variant compensation of resolution

The state of a space-variant degrading of an image is represented by a PSF parameter at each posi- tion in the image. Therefore, each resolution obtained by the wavelet transform must be degraded depending on the PSF parameter. Therefore, the value of correc- tion can be determined uniquely by relating the de- grading resolution and the PSF parameter. A space- variant correction of resolution is carried out by making a product of all the coefficients as follows:

II

where Flis the multiresolution image of a restored im- age; G) is the multiresolution image of the degraded image; Cf is the distribution of values of compensa- tion; i = -1, -2, -3, ...; and n = hh, hl, lh, 11. The restored image is obtained by applying a reversed wavelet transform.

Let us explain the method of determining the val- ue of the correction. It is assumed in this paper that the resolution data in each position in an image is de- graded in proportion to the data of the frequency band of each resolution. Then the resolution is corrected by using the distribution of reciprocal of the frequency of the PSF.

Figure 3 shows this relationship in a 2-dimen- sional illustration (for simplicity). First, the reciprocal of the frequency distribution of the PSF for a pixel in a blurred image is taken. All the frequency compo- nents which are corrected by this distribution must be treated at the same time, since the frequency axis in the multiresolution space is sampled roughly. There- fore, the value of the correction of resolution cannot be determined to be best for all the components. For example, a large correction value is suitable for a high- frequency region, but this will overcorrect a low-fre- quency region. Therefore, when the reciprocal wavelet transform is applied, the value of correction should be chosen so that the correction in each region is bal- anced.

In general, lower-frequency data of an image contain important information for the structure of an image. An overcorrection of data in a higher-fre- quency region generatesa ringing in its restored image. Therefore, to emphasize the data in the low-frequency region, the value of compensation of the resolution in this paper is taken from a Gaussian-weighted mean value as shown in Fig. 3. This calculation for a one- dimensional case is given by

where Ci is the distribution of the value of compen- sation at various resolutions; l/H(x) is the reciprocal of frequency distribution of PSF; w(u) is the distribution of a Gaussian weight; oN is a Nyquist frequency; and i = -1, -2, -3, ... .

78

blurred imaae /

w-w emoolhlng

blurred region

Fig. 4. Method for estimating a blurred region.

The number of pixels of a low-resolution image represented by a multiresolution model reduces one pixel in every four pixels of the original image. There- fore, let us determine the value of compensation of each pixel in a resolution by using the mean of re- covery values of pixels.

The mentioned processes are applied to all the pixels so that resolution of a space-variant blurred image is corrected depending on their positions.

3. Method of Estimating PSF in a Blurred Image

There are many cases where the data of a PSF of a blurred image are not available while the blurred image itself is given. It is very difficult to estimate the PSF from an arbitrary blurred image alone. Therefore, this paper is applied only to a case where a limited area is blurred (due to an imperfect focus) with a uniform blurring (for example, an object is out of focus with its correctly focused background). In such a case, after a blurred region has been estimated, the PSF parameter in this region is estimated.

3.1. Estimation of blurred region

The estimation is based on an assumption that an image is uniformly blurred in a limited region. When this image is decomposed into various resolutions by using a wavelet transform, variation of coefficients in

the region reduces due to the blurring. Therefore the blurred region is judged by the size of variance of the coefficients in a local reference region with a resolution of Wy(a large control by the blurring).

Figure 4 shows the flow of the method of esti- mating a blurred region. An image having a resolution of WY is enlarged to the size of the original image (N x N> by insulating pixels. Then this image is divided into reference regions of N, x N,, and the region in which the variance is less than a threshold is taken as a likely degraded region. This region is divided into four sections, and each section is examined for a likely region. This procedure is repeated so that smaller sections are obtained. When sections with 2 x 2 pixels are obtained, the blurred region among them is regard- ed as the finally likely region.

This procedure can avoid an erroneous recogni- tion of a region having no high-frequency component (e.g., the sky in a landscape) as a degraded region. Since a large region with no high-frequency compo- nent might be recognized erroneously as a degraded region, such as image is prediscriminated in this method.

3.2. Estimation of PSF parameter

The value of compensation of resolution in the proposed method is proportional to the PSF param- eter, since the degree of blurring is also proportional to this. If an estimated PSF parameter for an image restoration is too large, a stripe pattern (ringing) appears in the restored image due to an overcorrection of the coefficient. This suggests that a ringing in an image indicates an overestimate. To solve this prob- lem, many restored images with various PSF param- eters have been prepared, as shown in Fig. 5 . The PSF parameter is chosen from the value immediatelybefore a ringing starts. The occurrence of the ringing is determined when the number of pixels with minus val- ue (not really existing in an image) exceeds a threshold as shown in Fig. 5.

Figure 6 shows the flow of the proposed method. the wavelet transform is applied to a given blurred image. The blurred region and the SPF parameters in the region are estimated by high-resolution data. The data are corrected by multiplying the values of com- pensation calculated for each resolution. Then a re- stored image is obtained by applying the reverse- wavelet transform to the multiresolution image.

79

space-invariant restored image d=0.10.20.30.4 =-*.. 4.74.84.95.0

blurred image

L.

Fig. 5. Method for estimating PSF parameter.

Fig. 7. Blurred image dealt with in this study.

I blurred imaad

I estimation of (blurredl

I estimation of

I recovery of W.T.1

c

Fig. 6. Flow of the proposed method.

4. Examples of Applications

4.1. Applications to images blurred by computer

Space-variant blurred images (e.g., an object is out of focus with its correctly focused background) were synthesized by a computer to test the proposed method with the specified conditions. The image has 256 x 256 gray levels. The object in this image was blurred with a Gaussian PSF given by Eq. (3) using a PSF parameter of 2.5.

Figure 7(a) shows the original image, and Fig. 7@) shows its blurred image with the given conditions. Figure 7(c) shows the resolution of the image obtained by applying the wavelet transform. This shows that the coefficients for the object are intensely controlled by the degrading; Wl; was used for these examples.

Figure 8 shows examples of estimation of de- graded regions (the image of a person) using J#$. Figures 8(a) to (c) show the procedures of estimating a blurred region, gradually reducing the size from N = 16. This example shows a correct estimation in which a person's image is clearly extracted ignoring the backgrdund (a garden) which contains high-frequency components.

Figure 9 shows the relationship between the PSF parameter and the appearance rate, using the param- eter from 0.1 to 5.0 at 0.1 interval; 50 restored images were used for the estimation of the rate (half-maximum width). The appearance rate is the rate of the pixels with minus sign. This figure shows that the appearance rate suddenly increases at a PSF parameter of 2.5 pixels. It is regarded that the ringing occurrence when the PSF parameter becomes greater than 0.1, i.e., the half-maximum width in the blurred region is 2.6. This estimation is almost correct, although there are some incorrect estimations in the blurred region.

80

Fig. 8. Estimation of the blurred region.

Fig. 10. Restoration results of space-variant blurred image (1).

Fig. 9. Relationship between the PSF parameter and the appearance rate of the pixel with minus value.

Figure 10 shows examples of restored images using the estimated PSF parameter and the space-var- iant correction. Figure lO(a) shows the map of the value of compensation of each resolution using the estimated half-maximum width. Figure lo@) shows the corrected resolution. It is seen that the degree of the correction is different between the main subject and the background. Figure 1O(c) shows the restored image in which the main subject alone is fully restored.

Figure 10(d) shows a restored image by using a Wiener filter. This method overcorrects the main subject (human figure), and undercorrects its back- ground. This contrasts to Fig. 1O(c) which is restored by using the proposed method (space-variant correc- tion) so that a good balance is obtained throughout the image.

A restored image can be evaluated quantitatively bY

where Zi, j) is a restored image, and f(Z, j ) is the original image. The smaller the E, the closer the re- stored image to the original. Table 1 shows the values of E for the three methods, the proposed method being the best. Note that the Wiener method has a small E because this is not a space-variant treatment in addi- tion to the occurrence of some ringing.

81

Table 1. Evaluation values of restored image

Image I Evaluation value E (%) Blurred image Proposed method Wiener filter 13.81

Table 2. Evaluation values for images restored on vari- ous levels of resolution

-2

-3

-4

-5 Blurred image 1)

2.90

2.79

2.77

2.73

3.31

Fig. 11. Restoration results of space-variant blurred image (2).

If the data of the PSF in a degraded image are known, the proposed method can be applied to a re- storation of an image by using the PSF parameter in each position of the image. Figure 11 shows an exam- ple of such a case. Figure ll(a) shows an image in which the degree of blurring is gradually increased from left to right using a computer. Figure ll(b) shows the resolution of Fig. ll(a), indicating that the control is greater for a part where the degrade is greater (the right side of the image). Figure ll(c) shows the corrected resolution. Figure 1 l(d) shows the correctly restored image.

In the space-variant restoring of the image, the image was decomposed in a resolution of W-2 using the wavelet transform, and compensated by using the method described in section 2.3. If this decomposition is not sufficient, the restoration of an image becomes poor. This is because the decomposition in the fre- quency direction becomes inadequate, thus the restora- tion does not correspond to the PSF. Table 2 shows the evaluation values against resolutions (from W!’l to W;). The table also shows the excellence of the image with W_”, compared with other values. In the space- variant restoring of an image, the resolution should be decomposed adequately, and the correction should be appropriate for space and frequency. It is important that the wavelet transform functions correctly.

4.2. Applications to blurred image taken by CCD camera

Figure 12(a) shows a space-variant degraded image. This image was made by taking two photo- graphs (houses and trees) with a CCD camera. The photograph of the trees is placed farther than that of houses from the CCD camera so that the CCD camera correctly focuses only the image of the trees keeping the image of houses out of focus. Figure 12(b) shows the wavelet transform of Fig. 12(a). It is seen that the image of the houses is suppressed by the degrading of their data. Figure 12(c) shows the blurred region esti- mated by the proposed method. Note that the blurred region of the house is extracted almost correctly, al- though its boundary is not perfect. Figure 12(d) shows the finally restored image. Note that the blurred parts in the original image (e.g., roof tiles of the houses) are improved.

Figure 13 shows the relationship between the PSF parameter and the appearance rate. This was used to estimate the half maximum width of Fig. 12(c). In the image of the real landscape, the threshold for the ringing is chosen as large as 0.3 percent, since this

82

(a) SNR=U. I drj. ( I , ) S N H = - ~ X -2 , i u . Estimated PSF 2.4. Estimated PSF 2.1.

Fig. 14. Estimated blurred region of blurred image with noise.

which is more accurate due to less noise. Note that the PSF parameter (true value 2.5) is estimated as 2.6 when no noise is contained, and the accuracy of the estimation deteriorates with noise.

Fig. 12. Restoration results of blurred image taken The problem of the noise will be an important with CCD camera. subject to study using the proposed method.

0. I I I -/ Fig. 13. Relationship between the PSF parameter and

the appearance rate of the pixel with minus value.

kind of image contains noises. As a result, the half- maximum width of the degraded region was 2.8.

The result for the degraded image taken by the CCD camera is less successful than the synthesized image shown in section 4.1 due to the noise of the camera. Figure 14 shows the regions of the blurred images shown in Fig. 7(a). Compare this with Fig. 8(c)

5. Conclusions

This paper proposes a method of restoring a space-variant blurred image by applying the wavelet transform which treats the space-variant frequency data. The resolution data in a blurred image also is treated by the wavelet transform, and the frequency data are restored by multiplying the value of compen- sation depending on the PSF parameter.

The blind deconvolution (which has been thought to be difficult) has been realized by limiting the applications only to uniformly blurred images.

A sample of a blurred image produced by a com- puter, and a sample of a synthesized image taken with a CCD camera, were tested by the proposed method. For the synthesized image, the blurred region and the PSF parameters have been estimated successfully. By using the value of compensation, restored images with a high quality have been obtained. For the synthesized image with the CCD camera, the results are also good although they have a lower quality than the computed image. This is due to the error in the estimation of the PSF caused by noise during the taking of the image (i.e., there is little noise, the result would be better). The results all confirm the effectiveness of the pro- posed method.

83

The estimation of the PSF in this method is af- fected by noise. The method cannot be applied to an image in which data of the PSF parameter is degraded. It is necessary to improve the method so that this can withstand noises, and to obtain a better value of com- pensation.

Acknowledgement. The authors with to express their thanks to Prof. Masato Nakajima (Faculty of Science and Technology, Keio University) for his kind advice.

REFERENCES

1. M. Takaki and Shimoda. Image Analysis Hand- book. Tokyo University Press, pp. 383-423 (1991).

2. A. K. Katsaggelos (ed,). Digital Image Restora- tion. Springer-Verlag (1991).

3. M. 1. Sezan and A. M. Tekalp. Survey of recent developments in digital image restoration. Optical Engineering, 29,5, pp. 393-404 (1990).

4. S. Komatsu. Image recovery based on blind decon- volution. Collected Papers, Proceedings of the 22nd Joint Conference on Imaging Technology, Japan, pp. 179-184 (1991).

5 . K. Nishi. Problems and the improved method re- lated to the image restoration by means of convex projection. Collected Papers, Japan, pp. 189-192 ( 1991).

6. K. Nishi and S. Ando. Proposal of super-resolu- tion method with blind deconvolution. Technical Papers, I.E.I.C.E., Japan, IE92-1 (1992).

7. S. Hashimoto and Hide1 Saito. Method of restora- tion of shift-variant blurred images using estimated PSF parameter. Trans. I.E.I.C.E., Japan, J77-D-

8. H. Kituchi. Wavelet analysis of abnormal burning. Suri Kagaku, 36, 12; pp. 44-51 (1992).

9. M. Ohta. Application of wavelet to image coding. System/ControVInformation, Japan, 37, No. 7, pp.

10. M. Yamaguchi and A. Morimoto. Wavelet and its applications. [q/[II]. Journal of the Society of Instrument and Control Engineers, 8/10, pp. 879-

11. M. Sato. Mathematical bases of wavelet theory. Jour. of Japan Acoustical Society, 47, 6, pp. 405-

12. Y. Meyer. Wavelets and Its Applications. Spring- er-Verlag (1992).

13. S. G. Mallat. A theory for multiresolution signal decomposition: the wavelet representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11, No. 7, pp. 674-693 (1989).

14. S. G. Mallat. Multifrequency channel decompo- sition of images and wavelet models. IEEE Transactions, on Acoustics, Speech and Signal Processing, 37, 12, pp. 2091-2110 (1989).

15. M. Iwane. Estimation of distances in blurred images due to variation of aperture. Trans. I.E.I.C.E., Japan, J74-D-II, pp. 1192-1199.

16. L. R. Berriel, J. Bescos and A. Santisteban. Image restoration for a defocused optical system. Applied Optics, 22, No. 18, pp. 2772-2780 (1983).

II4, NO. 4, pp. 719-728 (1994).

402-409 (1993).

886/1066-1074 (1992).

423 (1991).

AUTHORS

Shoichi Hashimoto graduated from the Electrical Engineering Department, Keio University in 1993, and received his Master's degree from the same university in 1995. He joined "IT in 1995. He is engaged in research on image restoration as a graduate student.

Hideo Saito graduated from the Electrical Engineering Department, Keio University in 1987, received his Master's degree in 1992 and his Dr. of Eng. degree later. He was appointed Instructor in the same department in 1992 and an Assistant Professor in 1995. He has engaged in research on CT measurements, and image processing. He is a member of Society of Instrument and Control Engineering; Inst. of Information Processing, Japan; and IEEE.

84