defocus morphing in real aperture images

1MvhgpTeskcambt

spDstavhtomRpanwVebpbd

Subhasis Chaudhuri Vol. 22, No. 11 /November 2005 /J. Opt. Soc. Am. A 2357

Defocus morphing in real aperture images

Subhasis Chaudhuri

Department of Electrical Engineering, Indian Institute of Technology, Bombay, Powai, Mumbai 400 076, India

Received December 15, 2004; revised manuscript received March 30, 2005; accepted March 30, 2005

A new concept called defocus morphing in real aperture images is introduced. View morphing is an existingexample of shape-preserving image morphing based on the motion cue. It is proved that images can also bemorphed based on the depth-related defocus cue. This illustrates that the morphing operation is not necessar-ily a geometric process alone; one can also perform a photometry-based morphing wherein the shape informa-tion is implicitly buried in the image intensity field. A theoretical understanding of the defocus morphing pro-cess is presented. It is shown mathematically that, given two observations of a three-dimensional scene fordifferent camera parameter settings, we can obtain a virtual observation for any camera parameter settingthrough a simple nonlinear combination of these observations. © 2005 Optical Society of America

OCIS codes: 100.1830, 100.6890, 110.2990, 330.3790, 330.6110.

gimc(lr

tvgrpfiavvcm

ifiedsitdscaoeHfts

. INTRODUCTIONany research efforts have been reported in the area of

iew interpolation1 and view morphing.2 The basic ideaere is to generate a new view of a scene such that theeometric structure of the scene in the rendered view isreserved, given two or more different views of the scene.his has many practical applications in virtual view gen-ration. The key advantage of view morphing is that thecene structure does not have to be computed, albeit anowledge of camera parameters and the feature pointorrespondence is required. The new view can be gener-ted using the Beier and Neely algorithm,3 and the ho-ography relationship for the novel view can be shown to

e a convex combination of homographies of the respec-ive views.

Since the work of Seitz and Dyer,2 a large amount of re-earch literature is available that performs shape-reserving novel view generation. For example, Seitz andyer also derive the conditions under which a given set of

parse views of a scene is good enough to generate a con-inuous range of view points.4 Lhuillier and Quan explore

robust method to handle two main difficult tasks iniew morphing, namely, pixel matching and visibilityandling.5 Kang et al. discuss the trade-off issues relatedo the need for accurate three-dimensional (3-D) modelingf a scene in terms of geometry and photometry whileaintaining the photorealism of the rendered scene.6

athi et al. assume the surface to be composed of planaratches while generating the novel views.7 In Ref. 8 theuthors use the polygonal approach to map the texture inovel views. Avidan et al. suggest use of a trilinear tensorhile synthesizing widely separated novel views.9,10

azques et al. investigate the optimal positions for cam-ra placement to obtain a minimal set of views for image-ased rendering.11 Laveau and Faugeras12 address theroblem of generating a new view from a reference viewy interactively selecting four points in the scene to pre-ict a new view. Svedberg and Carlsson also attempt to

1084-7529/05/112357-9/$15.00 © 2

enerate a novel view from a single constrained scene us-ng the symmetry property.13 A general review on view

orphing can be found in Ref. 14. In all these cases, onehanges the viewpoint, and hence the camera geometrymostly the external parameters and in some cases theens-to-image plane distance also) plays an importantole.

The concept of view morphing has also been ex-ended to many other applications, such as dynamiciew interpolation,15–17 model-based interactive vieweneration,18 and image coding based on viewendering.19 Weiskopf et al.20 propose an interesting ap-lication of image-based rendering wherein the plenopticunction is appropriately modified to take care of relativ-stic aberration of light and searchlight effect to simulate

superfast motion film. Ezzat and Poggio21 applied theiew morphing technique in morphing visemes for text-to-isual speech synthesis. Radke and Richard emulate theoncept in audio interpolation to create a virtualicrophone.22

The view morphing technique assumes that the cameras a pinhole camera and hence it has an infinite depth ofeld. However, all commercial cameras have a finite ap-rture and hence provide only a limited depth of field. Theepth-related defocus23 also provides an important cue fortructure recovery. This also makes the image photoreal-stic and an object of delight to a connoisseur. Now we askhe question, can we also perform a structure-preservingefocus morphing of a scene? For example, given two ob-ervations of an arbitrary 3-D scene under two differentamera parameter settings, resulting in differentmounts of blurring in the pictures, can we obtain a novelbservation of the same scene for a virtual camera param-ter setting without estimating the depth in the scene?ere the possible camera parameters include aperture,

ocal length of the lens, and the lens-to-image plane dis-ance. It may be noted that researchers in computer vi-ion do not consider the aperture and the focal length of

005 Optical Society of America

ttptsitecetdeptsphatad

ppscmgteapcpvcp

wrtabitmthtKatlfpcr

ott

cedaS

2TttdEcnifiv

ALalfafbds

witarlhnpbttfiba

stps

w(ag

2358 J. Opt. Soc. Am. A/Vol. 22, No. 11 /November 2005 Subhasis Chaudhuri

he lens as part of the internal camera parameters sincehey consider a pinhole model. In real aperture photogra-hy, all of them play important roles. The novel view syn-hesis problem as discussed in Ref. 2 is a case of motion ortereo cue morphing, and, to be more precise, we may callt a plenoptic transfer. Similarly, the virtual setting of in-ernal camera parameters in real aperture images is anxample of defocus morphing. Hence the morphing pro-ess is no longer restricted to a change in camera geom-try alone—it may also involve photometric changes. Inhis paper it is proved that it is, indeed, possible to haveefocus morphing, and this problem is, in some sense,quivalent to view morphing. Just as in the view mor-hing technique where there are difficulties with respecto correspondence generation and the handling of occlu-ion, there will be equivalent problems with defocus mor-hing. Furthermore, for the view morphing problem theomography relationship for a novel view can be obtaineds a combination of those of the given views. We showhat the morphing with respect to the defocus cue follows

similar form of combination in terms of the observedata, although it is not a linear one.There is one basic difference between the view mor-

hing and the defocus morphing processes. In view mor-hing, the photometric properties of the scene are as-umed to be unchanged when the camera position ishanged. Thus one tries to account for disparity adjust-ents while doing the plenoptic transfer. This is purely a

eometric process. In the case of defocus morphing, sincehe optical center of the camera does not move, the geom-try remains unchanged (strictly speaking, there could besmall change in magnification when some of the cameraarameters are changed, which also needs to be ac-ounted for during defocus morphing). Hence one needs toerform only photometric adjustments between the twoiews to obtain the virtual observation. However, in bothases of image morphing, the 3-D structure of the scene isreserved.It is worth refering to the work of Kubota et al.,24–27

here the authors use a set of linear filters for photomet-ic adjustment of observations from a multifocus systemo obtain a fully focused, virtually rendered view. Theylso explain how visual special effects can be createdased on the depth-related defocus.26 The purpose of us-ng multifocus observations is the removal of aliasing inhe texture while rendering a new view as defocus re-oves the high-frequency components. My objective in

his paper is quite different in the sense that I explainow these multifocus obvervations themselves can be vir-ually obtained through defocus morphing. Further more,ubota et al. assume that the filters for each depth layerre known. I do not make any such assumption. Finally,heir method has the advantage of being a linear one un-ike the one in this paper, making an analysis of the per-ormance an easier proposition. However, the method pro-osed in this paper provides a better insight to theoncept of defocus morphing with respect to camera pa-ameters.

The organization of the paper is as follows. The conceptf defocus morphing is introduced in Section 2 along withhe necessary physical interpretation. Then it is shownhat the morphing process can be explained through the

hange of any of the following internal camera param-ters: aperture, focal length, and the lens-to-image planeistance. Some experimental results on defocus morphingre presented in Section 3. The paper concludes inection 4.

. VIRTUAL DEFOCUS RENDERINGhe following problem is considered: Given two observa-

ions E1 and E2 of the same scene under two different set-ings of camera parameters C1 and C2, we obtain a ren-ered view of the same scene as a function of only E1 and2 for any virtual camera parameter setting C that is a

onvex combination of C1 and C2. The imaging model iso longer a pinhole model. We consider the real aperture

maging principles. It is proved that one does not have tond the depth at each point in the scene to render theiew.

. Defocusing Processet us now look at the image formation process in a realperture camera employing a thin lens. When a pointight source is in focus, all light rays that are radiatedrom the object point and intercepted by the lens converget a point on the image plane. When the point is not inocus, its image on the image plane is no longer a pointut a circular patch of radius � that defines the amount ofefocus associated with the depth of the point in thecene. It can be shown23 that

� = �rv� 1

F−

1

v−

1

Z� , �1�

here r is the radius of the aperture, v is the lens-to-mage plane distance, F is the focal length of the lens, Z ishe depth (distance from the optical center) at that point,nd � is a camera constant that depends on the samplingesolution on the image plane (see Fig. 1 below for an il-ustration). If the aperture r=0, it corresponds to a pin-ole image when every point in the scene is in focus sig-ifying �=0. Let us use the symbol I�x ,y� to represent theinhole (focused everywhere) image of the scene. It maye noted that I�x ,y� is not a physically observable quan-ity in a real aperture camera when there is depth varia-ion in the scene. From Eq. (1) we note that C= �r ,F ,v� de-nes the internal camera parameters each of which maye changed to effect a different amount of defocus blur forfixed depth Z.The depth-related defocus process is linear but not

pace invariant. Assuming a diffraction-limited lens sys-em, the point-spread function of the camera system at aoint �x ,y� may be approximately modeled as a circularlyymmetric two-dimensional Gaussian function:28–30

h�x,y� =1

2��2exp�−x2 + y2

2�2 � , �2�

here the blur parameter ��x ,y� has been defined in Eq.1). Assuming the depth to be constant (we shall relax thisssumption later), the observed defocus image E�x ,y� isiven by

wtmb

ttirtG(

BLaEerbs

wEas

o

T

Withfsbtt�t

casIm(

wraeaSqae

Am

woncs

Htt�stifsn

tnf


E�x,y� = I�x,y� � h�x,y�, �3�

here the operator � represents convolution. It is easiero represent the above relationship in the spectral do-ain. Denoting the Fourier transform of a function f�x ,y�

y f��x ,�y�, we obtain

E��x,�y� = I��x,�y�h��x,�y� = I��x,�y�exp�−�2��x

2 + �y2�

2� .

�4�

It may be mentioned that if one follows geometric op-ics, then the blur kernel h�x ,y� has a circular shape andhe corresponding frequency response of the lens systems given by the Bessel function (see Hopkins31 for a theo-etical analysis and Baker32 for an experimental valida-ion). In this paper we restrict ourselves to using theaussian blur model valid for a diffraction-limited lens

see Ref. 28).

. Defocus Morphinget us now get back to the image morphing problem. Forgiven scene, one can have two defocused observations

1 and E2 corresponding to two different camera param-ter settings C1 and C2 such that the resulting blur pa-ameters are �1 and �2, respectively. Since the defocuslur is characterized by the parameter �2 in Eq. (4), let uselect the morphing relationship

�2 = ��12 + �1 − ��2

2, �5�

here 0��1 and �i2 corresponds to the blur in image

i. Why we select the convex combination in terms of �12

nd �22 and not �1 and �2 will be explained in the next

ubsection. From Eqs. (4) and (5) we obtain

E��x,�y� = I��x,�y�exp−1

2��1

2 + �1 − ��22��x

2 + �y2��

= I��x,�y�exp�−�1

2��x2 + �y

2�

2��

� I��x,�y�exp�−�2

2��x2 + �y

2�

2��1−�

,

r

E��x,�y� = E1��x,�y�E2

�1−��x,�y�. �6�

aking logarithms of both sides,

ln E��x,�y� = � ln E1��x,�y� + �1 − ��ln E2��x,�y�. �7�

hen we compare Eq. (7) with the corresponding render-ng equation for the view morphing problem in Ref. 2,hey are very similar except that the convexity relationolds good in the logarithmic scale after having trans-ormed the images into the spectral domain. Hence we ob-erve that any two defocused observations can be com-ined using the relationship given in Eqs. (6) to renderhe scene for a virtual camera parameter setting suchhat the resulting blurring parameter �2 lies in the range12��2��2

2. Further more, there is no need to calculatehe depth or structure in the scene.

The morphing relationship as given in Eqs. (6) does notonsider the presence of noise in observations. Let us nownalyze what happens if noise is present. Let Ni be theensor noise in the ith noisy observation denoted as Eni.n the Fourier domain, Eni= Ei+Ni. Let En be the result oforphing two noisy observations En1 and En2. Now Eqs.

6) can be written as

En��x,�y� = En1� ��x,�y�En2

�1−��x,�y�

= E1��x,�y� + N1��x,�y��E2��x,�y�

+ N2��x,�y��1−��

= E1��x,�y��1 +

N1��x,�y�

E1��x,�y��

E21−��x,�y�

��1 +N2��x,�y�

E2��x,�y��1−��

= E1��x,�y�1 + �1��x,�y��

�E21−��x,�y�1 + �2��x,�y��1−��,

here �1 and �2 are measures of inverse signal-to-noiseatio (SNR) at various spatial frequencies. Since thebove equation is nonlinear, it is difficult to quantify theffect of noise on the morphed image analytically. Thenalysis is done for two special cases, namely, when theNR is high and when it is very low. Typically at low fre-uencies, the SNR is quite high when �1 , �2 1. Thebove equation can be simplified (the arguments in thequations are also dropped for simplicity) to

En = E1�E2

1−��1 + ��1�1 + �1 − ��2�. �8�

ssuming the noise N1 and N2 to be independent and zeroean, it is easy to show that

E�En I� = E1�E2

1−� = E, �9�

here E is the expectation operator. Thus we observe thatne does not have much problem dealing with observationoise at lower frequencies. However, at higher frequen-ies, the SNR is small and hence �1 and �2 may not bemall compared with unity. Thus

E�En I� � E1�E2

1−�. �10�

ence the estimate of the high-frequency components inhe morphed image will be biased, resulting in a degrada-ion in the quality of the synthesized view. However, if �0 or 1, the linear approximation as shown in Eq. (8) is

till quite valid, signifying that as one moves away fromhe given two observations En1 and En2 while synthesiz-ng the virtual view, the effect of noise in the high-requency components may become noticeable. However,ince �� 0,1�, the effect of noise is minimal and can beeglected for all practical purposes.A few more comments are now in order. In Eqs. (6) the

erm E1 ��x ,�y� refers to the image enhancement tech-ique known as root filtering.33 It boosts up the high-requency components and thereby reduces the amount of

basafs�fitfist

EstvthodtscTt�c

ieTiltcwc

swt

CTppunr

cde

fl

Tacmafdd1cr

ftacoeepnmp

DFfistS

tpct

rtrivAass

Fbt


lurring. In case the observations E1 and E2 have a goodmount of sensor noise, the rendered image E�x ,y� mayuffer from a noise enhancement problem. However, were not exactly performing root filtering as the phase in-ormation is preserved in root filtering. But in the expres-ion E1

��x ,�y�, the phase is altered based on the value of. It is interesting to note that the structure of the rootlter (one should not confuse this with the square-root fil-ering technique used in subspace updating or Kalmanltering applications) is not well understood in the time/patial domain, and very little information is available inhe literature.

The derivation of the rendering relationship given inqs. (6) is obtained by assuming that the scene has a con-tant depth. Now we relax the above assumption. Whenhere is depth variation in the scene, Eq. (3) is no longeralid as the blurring process becomes shift variant. In aypical depth from defocus (DFD) problem, this issue isandled by forming a small M�M window about a pointver which the depth can be assumed to be constant. Theepth computation is done locally at the center pixelhrough Eq. (3), and a dense depth map is obtained byliding the window by a single pixel. The same procedurean be repeated while morphing the images E1 and E2.he morphing is performed at every point using the rela-ionship defined in Eqs. (6) by considering a small MM neighborhood over which the Fourier transforms are

omputed.A comparison with the view morphing process results

n an understanding that the knowledge of homography isquivalent to knowing the thin-lens camera parameters.he need to establish the feature point correspondences

n view morphing now translates into an equivalent prob-em of having to handle a shift-varying system. Similarly,he problems of holes and folds persist, as the defocus pro-ess is known to suffer from partial self-occlusion.34 Theider the aperture, the more pronounced the effect of oc-

lusion.Now that we have derived the blur morphing relation-

hip, we discuss the physical interpretation for morphingith respect to each of the camera parameters C and/or

he scene structure.

. Aperture Morphinghe defocus blur is a function of three different cameraarameters, namely, the aperture �r�, the lens-to-imagelane distance �v�, and the focal length �F� of the lenssed. It is also a function of the depth �Z� in the scene. Weow look at the physical interpretation of the morphingelationship �2=��1

2+ �1−��22. Assuming that all other

ig. 1. Illustration of aperture morphing. The blur is increasedy increasing the aperture. Aperture morphing is possible withinhe range �r ,r �.
1 2
amera parameters remain constant, using Eq. (1) thatefines the blur radius � in terms of the camera param-ters, we get

r2 = �r12 + �1 − ��r2

2. �11�

Multiplying both sides by � /2 and noting that this re-ers to the aperture area A for the lens, the morphing re-ationship can be written as

A = �A1 + �1 − ��A2. �12�

hus if the observations for two given aperture areas A1nd A2 are given, an image for any aperture A1�A�A2an be synthesized. This also explains why we started ourorphing relationship in terms of �2 and not �. The

mount of light collected at the image plane is a linearunction of the aperture area �A� and not the aperture ra-ius. The basic morphing process with respect to blurringue to aperture change is illustrated in Fig. 1. From Fig.it is clear that any virtual aperture in the range �r1 ,r2�

an be synthesized given the observations E1 and E2 cor-esponding to the apertures r1 and r2, respectively.

If one changes any of the camera parameters, apartrom the change in blur radius, there are other effects inhe captured image such as the magnification. Because ofchange in aperture area, there is no change in magnifi-

ation as the equivalent pinhole geometry (i.e., the pointf convergence of the incident rays) is preserved. How-ver, the image plane receives a different amount of lightnergy that is usually compensated in a camera by appro-riately changing the shutter time. Since this option isot available while morphing these images, we recom-end a renormalizing operation of gray levels of the mor-

hed image based on the input images.

. Lens-to-Image Plane Distance Morphingor the given blur parameters �1

2 and �22, we now want to

nd out what would be the equivalent morphing relation-hip in terms of corresponding lens-to-image plane dis-ances v1 and v2, instead of a variation in the aperture.ince in the expression

�i = �rvi�1/F − 1/Z� − 1� �13�

he term �1/F−1/Z� refers to the actual lens-to-imagelane distance for which the image will be sharply fo-used, we denote this term as 1/vo. The morphing rela-ionship �2=��1

2+ �1−��22 now becomes

�v12 + �1 − ��v2

2 − v2 = 2vo�v1 + �1 − ��v2 − v�. �14�

On simplification, this takes a nice form

�v − vo�2 = ��v1 − vo�2 + �1 − ��v2 − v0�2. �15�

The above relationship closely resembles the morphingelationship used in this study. The only departure is thathe lens-to-image plane distances should be measuredelative to the actual distance v0 for which the scene points perfectly in focus. This is now illustrated in Fig. 2. Airtual image plane can be synthesized between the lines1B1 and A2B2 given the corresponding observations E1nd E2. From Fig. 2 we observe that the above relation-hip can also be easily derived using the properties ofimilar triangles. The same thing can be said about Eq.

(i(w

sm(fis

EIfftao

tfpa

o

olffcw

tsogwto

FIiesiemlatrctwaopmtatt

sf

Fv

Ft

FrsA


11), the one used for aperture morphing, while obtainingt from the basic blur morphing relationship given in Eq.7). Hence the physical significance is the same as whatas discussed in the previous subsection.It is to be noted that, since the aperture remains con-

tant, there is no need for camera gain adjustment whileorphing. However, there is a change in magnification

see Fig. 2). Thus the images E1 and E2 should be magni-cation adjusted while morphing them. A linear isotropiccaling achieves the purpose.

. Focal-Length Morphingn the DFD problem, one has the option of changing theocal length �F� of the lens to obtain the depth-related de-ocus. Let us now find out the equivalent morphing rela-ionship in terms of the focal lengths of the two lenses (F1nd F2) for which the observations E1 and E2 have beenbtained.

For a given depth �Z� and the lens-to-image plane dis-ance �v�, let F0 be the focal length of a lens that wouldocus the object sharply on the image plane. The mor-hing relationship �2=��1

2+ �1−��22 can now be written

s

� 1

F−

1

F0�2

= �� 1

F1−

1

F0�2

+ �1 − �� 1

F2−

1

F0�2

,

r

�F0

F− 1�2

= ��F0

F1− 1�2

+ �1 − ��F0

F2− 1�2

. �16�

ig. 2. Illustration of lens-to-image plane distance morphing. Airtual image plane between A1B1 and A2B2 can be constructed.

ig. 3. Illustration of focal-length morphing. A virtual lens withhe focal length in the range �F1 ,F2� can be constructed.

Hence the equivalent focal-length morphing dependsn the squares of the harmonic difference of the focal-ength setting of the observations with respect to the trueocal length needed to have the image captured in sharpocus. This case is illustrated graphically in Fig. 3. Onean construct a virtual lens having a focal length any-here between F2 and F1.The physical significance of the above relationship is

hat the point of convergence of all rays emanating from acene point changes due to the change in the focal lengthf the lens. The amount of defocus depends on the diver-ence of the beam from the focused imaged point. Like-ise in the previous case, there is a change in magnifica-

ion that needs to be accounted for while morphing thebservations.

. Depth Morphingn Subsections 2.B–2.D we explained the defocus render-ng process with respect to the virtual settings of the cam-ra parameters. Now we look at a different hypotheticalituations that would also let us morph the observationsn precisely the same manner. We assume that the cam-ra parameters are unchanged but the object can beoved toward or away from the camera (see Fig. 4 for il-

ustration). To keep the geometry unchanged, i.e., the im-ge of the point in the scene should remain unchanged,he point in the scene should be moved along the principalay (the line ABCO joining the scene point and the opticalenter) as illustrated in Fig. 4. Here the morphing opera-ion allows us to virtually move a point in the scene any-here between the line segment AB, where the points And B denote the location of the same scene point in twobservations E1 and E2, respectively. The movement of aoint along the line ABCO demands that the image beagnified as the depth increases so that the camera con-

inues to observe the same part of the scene through thedjustment of its field of view. By substituting Eq. (1) inhe blur morphing relationship defined in Eq. (5), we ob-ain

� 1

F−

1

v−

1

Z�2

= �� 1

F−

1

v−

1

Z1�2

+ �1 − �� 1

F−

1

v−

1

Z2�2

.

Using the basic lens equation, where Z0 denotes thecene depth for which the object would have been in per-ect focus on the image plane, we obtain

�1 −Z0

Z�2

= ��1 −Z0

Z1�2

+ �1 − ��1 −Z0

Z2�2

. �17�

ig. 4. Illustration of depth morphing. For a fixed camera pa-ameter setup and for the principal ray ABCO, a point in thecene can be virtually moved anywhere between the line segmentB.

tpetmt

p

capp

3Avassoftd

wtpo

dadsiFwnts�ltgiwtwOri

vgbaapv

Fsbl

Fug


The above expression is quite similar to what was ob-ained in the previous sub section on focal-length mor-hing, i.e., the depth morphing follows a harmonic differ-nce relationship. Thus, given the pictures of a scene atwo different depths, the scene can be rendered with geo-etric correctness for an intermediate depth provided

hat the pictures are adjusted for scaling changes.Thus we summarize that the basic defocus blur mor-

hing relationship

E��x,�y� = E1��x,�y�E2

�1−��x,�y�

an be explained in terms of morphing through changes inny of the camera parameters. All these equivalent mor-hing relationships given in Eqs. (11), (15), (16), and (17)roduce the same effect of morphing on the observed data.

. EXPERIMENTAL RESULTSlthough the primary motivation in this paper is to de-elop a mathematical theory on virtual camera parameterdjustment, I do provide some relevant experimental re-ults for illustration. In this section I show the results ofhape-preserving defocus morphing given two arbitrarybservations. For brevity I present the results for only aew values of the morphing parameter �. Experimenta-ions have been performed on both real and simulatedata sets.Figures 5(a) and 5(b) show the observations of a scene

here there is a large amount of depth variation, underwo different settings of the camera parameters. The firsthotograph [Fig. 5(a)] is better focused than the secondbservation [Fig. 5(b)]. Because of a large change in the

ig. 5. (a) and (b) Views of an audience during a seminar pre-entation under two different focusing adjustments. Results oflur morphing with �=0.5 when performed (c) globally and (d)ocally.

epth field, different parts of the scene have differentmounts of defocus. Clearly, an assumption of a constantepth in the scene is not valid and hence the morphinghould not be done globally based on the expression givenn Eqs. (6). The corresponding result for �=0.5 is given inig. 5(c) and one can see a fudge pattern almost every-here in the image. This is due to the fact that Eqs. (6) isot valid for a scene having depth variations. However,he locally morphed image that does not assume a con-tant depth in the scene with a neighborhood size of 88, as shown in Fig. 5(d), does not suffer from this prob-

em. The penalty for obtaining the improved result is inerms of increased computation. The order of computationoes up from O�N2 log N� to O�N2M2 log M� where N�Ns the size of the image and M�M is the window size overhich the depth is assumed to be constant. As expected,

he amount of blurring in the morphed image is some-here in between those in the two given observations.ne can render the same scene for any other camera pa-

ameter setting by suitably adjusting the parameter � us-ng one of Eqs. (11), (15), and (16), or (17).

The experiments are repeated for another set of obser-ations given in Figs. 6(a) and 6(b). Here there is aradual variation of depth in the scene. The results of glo-al and local morphing for �=0.7 are given in Figs. 6(c)nd 6(d), respectively. Note that the globally morphed im-ge that assumes a constant depth in the scene is not asoor as it was in the previous example due to a smallariation in the scene depth. However, the results of the

ig. 6. (a) and (b) Views of a scene with slowly varying depthnder different camera parameter settings. (c) and (d) Results oflobal and local morphing for �=0.7.

lipdcdwtu

pmFdfi[oiotuavtrptgtsdts

rsHkmwfacvmovpwFim

Fttem

FuaR(ap


ocal morphing method is definitely of better quality evenn this experiment. Hence we perform a structure-reserving blur morphing without actually estimating theepth in the scene. Both of the above experiments werearried out by changing the camera aperture to effect theepth-related defocus in the observations. Hence thereas no need for scale (magnification) adjustment. The in-

ensity variation due to aperture changes was removedsing gray-scale normalization in the images.The purpose of the next experiment is to evaluate the

erformance of the proposed morphing operation byatching the synthesized view to an actual observation.igures 7(a) and 7(b) show two views of a scene for twoifferent settings of the lens-to-image plane distance. Therst view is [Fig. 7(a)] poorly focused, and the second viewFig. 7(b)] is also partly defocused. Figure 7(c) shows an-ther image captured for a camera parameter setting thats somewhere in between the values used for the first twobservations. Can the image shown in Fig. 7(c) be ob-ained from those given in Figs. 7(a) and 7(b)? Since anncalibrated camera is used, what value of � will gener-te this particular view is not known. Hence all possibleiews for �� 0,1� are synthesized, in steps of size 0.1 andhe corresponding peak SNR (PSNR) is calculated withespect to the desired view. The corresponding peak SNRlot is shown in Fig. 8(f) and we find that �=0.2 provideshe best match. There is a clear peak in the plot that sug-ests that the synthesized view matches well with the in-ended view. The corresponding synthesized view ishown in Fig. 7(d). A comparison of these two imagesemonstrates that the synthesized view is quite close tohe true observation even in the presence of existing sen-or noise.

ig. 7. (a) and (b) Two views of a scene under different focus set-ings. (c) Same scene captured with a focus setting lying in be-ween the above two observations. (d) Result of synthetically gen-rating the view in (c) using images (a) and (b) through blurorphing.

The purpose of the next experiment is twofold. Theeader may remember that we have assumed a Gaussian-haped blur kernel to derive the morphing relationship.owever, a few researchers prefer to use the circular blurernel as obtained from geometric optics.23 In this experi-ent we study the validity of the Gaussian assumptionhen the blurring is indeed circular. Furthermore, the de-

ocus morphing relationship given in Eqs. (6) implicitlyssumes that we are working on the same side of the blurone as explained in Fig. 2, i.e., all rays are either con-erging or diverging. Thus the amount of bluring changesonotonically between the observations for 0��1, and

ne cannot perform deblurring while synthesizing a newiew. We also demonstrate this property through this ex-eriment. We synthetically generate two views of a sceneith varying depth using the Povray rendering toolkit.or the first view shown in Fig. 8(a), we adjust the focus-

ng distance to be somewhere in front of the nearest (left-ost) pawn. Hence the leftmost pawn is partly blurred

ig. 8. (a) and (b) Two synthetically rendered views of a scenender different aperture settings. (c) Another rendered view forn aperture lying in between the above two settings. (d) and (e)esults of synthetically generating the view in (c) using images

a) and (b) through blur morphing. (f) The plot of peak SNRgainst the morphing parameter � for both the current and therevious experiment.

bvsptrcbatdSmaniTspmccs[ivpi�vrtvwcpvotscc

4IctscltttbTtscmvl

achdm

AFtf

R

1

1

1

1

1

1


ut the distant pawn is poorly in focus. For the secondiew in Fig. 8(b), the focusing distance is adjusted to beomewhere in between the left pawn and the middleawn. The left and the middle pawns are relatively in bet-er focus compared with the distant pawn. Quite natu-ally, the observations correspond to same side of the blurone (diverging rays) for the middle and the right pawnsut correspond to both sides of the cone [(convergent in (a)nd divergent in (b)] for the left pawn. The aperture inhis experiment has been kept fairly large to reduce theepth of field, making the defocus blur quite prominent.ince the Povray toolkit uses a standard ray-tracingethod to render the scene, it follows the geometric optics

nd hence the defocus blur is circular in shape. Note thato noise is added to the rendered images other than what

s inherent during 8 bit quantization of the intensity field.he third view of the same scene shown in Fig. 8(c) is nowimulated with the focal distance so adjusted that thearallel rays converge at a point just in front of the left-ost pawn, signifying that the leftmost pawn is quite

lose to the second observation but the distant pawn isloser to the first observation. An attempt is now made toynthesize this view from the given two observationsFigs. 8(a) and 8(b)]. Various values of � in the range [0, 1]n steps of 0.1 are tried, and the quality of the synthesizediew against the given observation in Fig. 8(c) is com-ared by plotting the PSNR. From the PSNR plot shownn Fig. 8(f), we observe that there are two peaks—one at=0.2 and the other at �=0.8. The corresponding virtualiews are shown in Figs. 8(d) and 8(e), respectively. Theeason we get two peaks instead of a single peak is thathe synthesized views are not very close to the intendediew in Fig. 8(c). In the first case, the left pawn matchesell with that in the intended view, and in the second

ase the distant pawn matches well. Hence there are twoeaks. Since the assumption of monotonicity of the blur isiolated in two views, there does not exist a single valuef �, constant over the entire image, that will generatehe given view. On the other hand, it is found that the as-umption of a Gaussian-shaped blur kernel in place of aircular blur does not create much of a problem, as the re-onstruction of the individual pawns is quite good.

. CONCLUSIONSn this paper the basic concept of shape-preserving defo-us morphing has been introduced. There is no need to es-imate the structure in the scene. The morphing relation-hips in both cases of viewpoint change or internalamera parameter change have been shown to be equiva-ent; in the view morphing problem the homographies areo be morphed, while in defocus morphing the image in-ensity fields are to be morphed appropriately. In additionhe physical interpretation of the morphing process haseen presented in terms of various camera parameters.he primary emphasis in this paper has been in provinghe existence of an equivalent class of morphing relation-hips, be it geometric or photometric. Although the defo-us morphing does not generate the same kind of dra-atic effect as the view morphing does, it is nonetheless

ery important in understanding how to restore and ana-yze real-aperture images. Currently, my colleagues and I

re looking for practical applications of the morphing pro-ess in real aperture imaging. We are also investigatingow all these different types of cues, namely, motion andefocus, can be combined to have a simultaneousultiple-cue morphing.

CKNOWLEDGMENTSinancial assistance in the form of a research grant under

he Swarnajayanti Fellowship and programming helprom P. N. Vinay are gratefully acknowledged.

The author’s e-mail address is [email protected].

EFERENCES1. S. E. Chen and L. Williams, “View interpolation for image

synthesis,” in Proceedings of SIGGRAPH 93, Anaheim,California, August 1993 (www.siggraph.com), pp. 279–288.

2. S. M. Seitz and C. R. Dyer, “View morphing,” inProceedings of SIGGRAPH 96, New Orleans, Lousiana,August 1996 (www.siggraph.com), pp. 21–30.

3. T. Beier and S. Neely, “Feature-based imagemetamorphosis,” in Proceedings of SIGGRAPH 92,Chicago, Illinois, July 1992 (www.siggraph), pp. 35–42.

4. S. Seitz and C. R. Dyer, “View morphing: uniquelypredicting scene appearance from basis images,” inProceedings of Image Understanding Workshop, NewOrleans, Lousiana, 1997, pp. 881–887.

5. M. Lhuillier and L. Quan, “Image interpolation by jointview triangulation,” in Proceedings of the Conference onComputer Vision and Pattern Recognition, Fort Collins,Colorado, June 1999 (IEEE Computer Society, 1999), Vol. 2,pp. 139–145.

6. S. B. Kang, R. Szeliski, and P. Anandan, “The geometry-image representation tradeoff for rendering,” in Pro-ceedings of the International Conference on ImageProcessing, Vancouver, B.C. September 2000, Vol. 2, 13–16.

7. D. Rathi, G. Agarwal, P. Kalra, and S. Banerjee, “A systemfor image based rendering of walk-throughs,” inProceedings of Computer Graphics International(CGI2002), Bradford, UK, July 2002.

8. P. Debevec, G. Borshukov, and Y. Yu, “Efficient view-dependent image-based rendering with projective texture-mapping,” in 9th Eurographics Rendering Workshop,Vienna, Austria, June 1998.

9. S. Avidan, T. Evgeniou, A. Shashua, and T. Poggio, “Image-based view synthesis,” Artificial Intelligence (AI) Lab, AIM-1603 (MIT, 1997), CBCL paper-145.

0. S. Avidan and A. Shashua, “Novel view synthesis in tensorspace,” in Proceedings of Computer Vision and PatternRecognition, Puerto Rico, June 1997 (IEEE ComputerSociety, 1997), pp. 1034–1040.

1. P. Vazques, M. Feixas, M. Sbert, and W. Heidrich, “Imagebased modeling using viewpoint entropy,” in Proceedings ofComputer Graphics International CGI 2002, Bradford, UK,July 2002.

2. S. Laveau and O. D. Faugeras, “3D scene representation asa collection of images and fundamental matrices,” Tech.Report 2205 (Institut National de Recherche enInformatique et en Automatique, 1994).

3. D. Svedberg and S. Carlsson, “Calibration, pose and novelviews from single images of constrained scenes,” inProceedings of the 11th Scandinavian Conference on ImageAnalysis (SCIA’99), Kangerlussuaq, Greenland, June 1999,pp. 111–117.

4. Z. Zhang, “Image-based geometrically-correct photorealisticscene/object modeling (IBPhM): a review,” in Proceedings ofthe Asian Conference on Computer Vision (ACCV 98), HongKong, January 8–11, 1998, Vol. II, pp. 340–349.

5. S. Vedula, S. Baker, and T. Kanade, “Spatio-temporal view

1

1

1

1

2

2

2

2

2

2

2

2

2

2

3

3

3

3

3


interpolation,” in Proceedings of the 13th ACMEurographics Workshop on Rendering, SaarBruecken,Germany, 2002, pp. 1–11.

6. S. Baba, H. Saito, S. Vedula, K. M. Cheng, and T. Kanade,“Appearance-based virtual view generation for fly throughin a real dynamic scene,” in Proceedings of the JointEurographics and IEEE TVCG Symposium onVisualization, Interlaken, Switzerland, 2000, pp. 179–188.

7. R. A. Manning and C. R. Dyer, “Interpolating view andscene motion by dynamic view morphing,” in Proceedings ofthe Conference on Computer Vision and PatternRecognition, Fort Collins, Colorado, June 1999 (IEEEComputer Society, 1999), Vol. 1, pp. 388–394.

8. D. Cohen-Or, “Model-based view-extrapolation forinteractive VR web-systems,” in Proceedings of ComputerGraphics International (CGI97), Belgium, June 1997, pp.104–112.

9. M. Magnor, P. Ramanathan, and B. Girod, “Multi-viewcoding for image-based rendering using 3-D scenegeometry,” IEEE Trans. Circuits Syst. Video Technol. 13,1092–1106 (2003).

0. D. Weiskopf, D. Kobras, and H. Ruder, “An image-basedapproach to special relativistic rendering,” in Proceedingsof IEEE Visualization Conference October 2000 (IEEEPress, 2000), pp. 303–310.

1. T. Ezzat and T. Poggio, “Visual speech synthesis bymorphing visemes,” Int. J. Comput. Vis., 38, 45–57 (2000).

2. R. J. Radke and S. Rickard, “Audio interpolation,” inProceedings of the International Conference on Virtual,Synthetic and Entertainment Audio, Espoo, Finland, June2002.

3. S. Chaudhuri and A. N. Rajagopalan, Depth from Defocus:A Real Aperture Imaging Approach (Springer-Verlag, 1999).

4. A. Kubota, K. Takahashi, K. Aizawa, and T. Chen, “All-focused light field rendering,” in Proceedings of the

Eurographics Symposium on Rendering, Norrköping,Sweden, 2004, pp. 235–242.

5. A. Kubota and K. Aizawa, “Inverse filters forreconstruction of arbitrarily focused images from twodifferently focused images,” in Proceedings of IEEEInternational Conference Image Processing, Vancouver,2000 (IEEE Press, 2000), pp. 101–104.

6. K. Aizawa, K. Kodama, and A. Kubota, “Producing objectbased special effects by fusing multiple differently focusedimages,” IEEE Trans. Circuits Syst. Video Technol. 10,323–330 (2000).

7. A. Kubota, K. Aizawa, and T. Chen, “Virtual view synthesisthrough linear processing without geometry,” inProceedings of IEEE International Conference on ImageProcessing, Singapore, 2004 (IEEE Press, 2004), Vol. 5, pp.3009–3012.

8. M. Born and E. Wolf, Principles of Optics (Pergamon,1965).

9. A. P. Pentland, “A new sense for depth of field,” IEEETrans. Pattern Anal. Mach. Intell. 9, 523–531 (1987).

0. M. Subbarao, “Parallel depth recovery by changing cameraparameters,” in Proceedings of Second IEEE InternationalConference on Computer Vision Florida, December 1988(IEEE Computer Society, 1998), pp. 149–155.

1. H. H. Hopkins, “The frequency response of a defocusedoptical system,” Proc. R. Soc. London, Ser. A 231, 91–103(1955).

2. L. R. Baker, “An interferometer for measuring the spatialfrequency response of a lens system,” Proc. Phys. Soc.London, Sect. B 68, 871–880 (1955).

3. A. K. Jain, Fundamentals of Digital Image Processing(Prentice-Hall, 1988).

4. S. Bhasin and S. Chaudhuri, “Depth from defocus inpresence of partial self occlusion,” in Proceedings of IEEEInternational Conference on Computer Vision, Vancouver,Canada, July 2001 (IEEE Press, 2001), pp. 488–493.

defocus morphing in real aperture images

Documents