Vis Comput (2017) 33:1357–1369DOI 10.1007/s00371-016-1231-2

ORIGINAL ARTICLE

Kinetic depth images: flexible generation of depth perception

Sujal Bista1 · Ícaro Lins Leitão da Cunha2 · Amitabh Varshney1

Published online: 6 May 2016© Springer-Verlag Berlin Heidelberg 2016

Abstract In this paper we present a systematic approach tocreate smoothly varying images from a pair of photographsto facilitate enhanced awareness of the depth structure of agiven scene. Since our system does not rely on sophisticateddisplay technologies such as stereoscopy or auto-stereoscopyfor depth awareness, it (a) is inexpensive and widely acces-sible, (b) does not suffer from vergence - accommodationfatigue, and (c) works entirely with monocular depth cues.Our approach enhances the depth awareness by optimizingacross a number of features such as depth perception, opti-cal flow, saliency, centrality, and disocclusion artifacts. Wereport the results of user studies that examine the relationshipbetween depth perception, relative velocity, spatial perspec-tive effects, and the positioning of the pivot point and usethemwhen generating kinetic-depth images. We also presenta novel depth re-mapping method guided by perceptual rela-tionships based on the results of our user study. We validateour system by presenting a user study that compares the out-put quality of our proposed method against other existingalternatives on a wide range of images.

Keywords Kinetic depth effect · Depth perception ·Aesthetics in visualization · Visualization for the masses

B Sujal Bistasujal@umiacs.umd.edu

1 Department of Computer Science and Institute for AdvancedComputer Studies, University of Maryland, College Park,USA

2 Universidade Federal Rural de Pernambuco - UnidadeAcademica de Garanhuns, Garanhuns, PE, Brazil

1 Introduction

The kinetic-depth effect (KDE) is the perception of thethree-dimensional structure of a scene resulting from a rotat-ing motion. First defined by Wallach and O’Connell [53],the kinetic-depth effect has been used widely. Images thatexhibit KDE are commonly found online. These imagesare variously called Wiggle images, Piku-Piku, Flip images,animated stereo, and GIF 3D. We prefer to use the termKinetic Depth Images (KDI), as they use the KDE to give theperception of depth. In 2012, Gizmodo organized an onlinecompetition to reward the KDI submission that best provideda sense of depth [55]. Recently, the NewYork Public Librarypublished a collection of animated 3D images online [18].Flickr has a large number of groups that discuss and postanimated 3D stereo images. Also, online community-basedgalleries that focus on KDI can be found at the Start3Dwebsite [8]. With the increasing availability of stereo andlightfield cameras (such as the Lytro), the use of the KDE toexpress depth on ordinary displays is rising rapidly.

Although the basic form of the KDE is trivial to imple-ment for a virtual environment where the 3D geometry andlighting are known, excessive motion and reduced depthperception mar the visual experience unless proper care istaken. In fact, accomplishing a high-quality KDE from apair of photographs is not trivial and has several interestingnuances, as outlined in this paper, which should be useful forany practitioner wishing to use this technique for facilitatingdepth awareness. In this paper we describe an algorithm andits accompanying system, for facilitating depth awarenessthrough KDE using only a pair of photographs or images.

The use of the KDE is a viable alternative to the use ofstereoscopic and autostereoscopic displays: (i) the KDE pro-videsmonocular depth cues that allowus to experience depth,even with one eye, which accommodates people who suffer

123

1358 S. Bista et al.

from various monocular disorders, (ii) the depth is perceiveddue to the rotation of the object and does not require any spe-cial device (glasses or lenses) to view; this works with anydisplay and can be easily shared online, (iii) unlike stereo-scopic and auto-stereoscopic displays, KDE animations donot suffer from vergence-accommodation conflicts, and (iv)depth perception achieved by the KDE can exceed that ofthe binocular depth perception as the disparity provided bybinocular vision is limited compared to the angular rotationthat can be produced by KDE [12,35].

Although there are a large number of KDE-based imagesonline, most of them are manually created by the artist,require tedious user input to create them, or suffer fromvisualartifacts introduced by the automated systems used to createthem.Manual creation of these images may seem simple, butit is difficult for average users to make their own high-qualityKDE-based images. A lot of KDI suffer from artifacts causedby abrupt changes in motion, color, or intensity, as well asalignment errors and excessive motion.

The following are our contributions to this paper:

1. Given a stereo image or an image/depth pair as an input,we automatically optimize a KDE-based image sequenceby taking human depth perception, optical flow, saliency,radial components, and disocclusion artifacts into con-sideration;

2. We report the results of a user study that examines therelationship between depth perception, relative velocity,spatial perspective effects, and the positioningof the pivotpoint and use them to generate KDI;

3. We present a novel depth re-mapping method guided byimage saliency and the perceptual relationship found inour user study to reduce excessive motion in KDI.

2 Background

The kinetic depth effect (KDE) is defined as the percep-tion of the three-dimensional structural form of an objectwhen viewing it in rotational motion [12,53]. As an observermoves, the nearby objects are seen from different angles.To experience KDE, the optical-flow-driven motion cue hasbeen found to be very important [14,43,46] and can be expe-rienced from just two photographs taken fromdifferent views[11,26]. Another effect that gives the perception of depth isthe stereo kinetic effect (SKE). Although SKE andKDE bothgive the perception of depth, they are quite different. SKEgives a perception of depth when viewing a 2D pattern as itis rotated in the view plane (fronto-parallel) [41], whereasin KDE the depth perception arises from rotating the objectalong an axis. Motion parallax is another monocular depthcue that is experienced through the relative motion of thenear and far objects [12]. Generally, KDE is associated with

the rotational viewing of objects whereas motion parallax isassociated with the translational viewing of objects. Whenan observer fixates on an object and makes small rotationalor translational motions, both KDE and motion parallax aresimilar [12].

3 Related work

Although a few tools are available online to generate theKDEfrom image pairs, we found them to be largely ad-hoc tech-niques with variable quality. We have not come across anyprevious work in the technical literature that systematicallyidentifies the various competing constraints in building suchtools to achieve an aesthetically superior viewing result. Wenext give an overview of the various online tools we havecome across on the Internet.

The New York Public Library has an online tool, Stere-ogranimator, that converts stereograms into animated 3Dimages employing user input to align images [18]. Thetool allows users to manually rotate and translate stereoimages to align them and change animation speed to cre-ate animated GIF images that switch between images. Withcarefully acquired stereo images and with proper user input,the output of this tool can produce good results. Other-wise, the output may contain artifacts created by the abruptchanges in motion, color, or intensity, or through alignmenterrors.

Wiggle images follow the same principle and are cre-ated by taking a pair of stereo images (or a sequence ofimages) and flipping between them. With careful considera-tion during the acquisition stage and a very precise manualalignment during the animation generation stage, a reason-ably good quality effect can be achieved. However, theprocess is tedious and requires significant skill in photogra-phy and image processing. Rather than relying on the numberand quality of input images, we use a judicious mix of com-puter vision and rendering algorithms to create high-qualityanimations needed to experience the KDE from a pair ofimages.

Piku-Piku images from Start3D (www.start3d.com) pro-vides an online tool for generating KDI. A user uploads animage pair to the Start3D server, which then generates asequence of intermediate images and provides a hyperlink toview the resulting images on its server [8]. To the best of ourknowledge, the underlying algorithm for generation of Piku-Piku images, their processing, and their viewing has not beenpublished. A careful study of the animation created by Piku-Piku images shows that the intermediate frames between aninput pair of stereo images are computed as a translation fromone input image to other. This method does not produce agood results if the input set of stereo images does not happento fall on the path suited for the KDE. Blending artifacts are

123

Kinetic depth images: flexible generation... 1359

often seen and if the stereo images are not carefully acquired(for example, if shear or slant is present in the stereo pair),then the output is often of a poor quality.

Another similar tool is the Stereo Tracer created by Tri-axes (www.triaxes.com). This uses an image-depth pair ora stereo-image pair to generate intermediate views to createanimation for producing the KDE. To secure a good outputtedious manual adjustment is required. These adjustmentsinclude (a) depth map adjustment, (b) image alignment, and(c) manipulation of parameters that control parallax and theplane of zero parallax.

Recently, LytroCamera announced a perspective shift fea-ture on their camera application [32]. Based on user input,they allow users to change the camera perspective computedusing the captured light-field data. To our knowledge, theunderlying algorithm has not been published.

Zheng et al. [59] presented a method that focuses on auto-matically creating a cinematic effect that exhibits the parallaxeffect. They reconstruct the scene from a series of images,fill the occluded regions, and render the scene using a cam-era path that attempts to maximize the parallax effect whiletaking into account occluded regions. Rather than trying torecreate cinematic effect, we try to maximize depth percep-tion based on KDE while reducing motion-induced artifacts.KDE uses rotational motion that is different from the cine-matic effects presented by Zheng et al. [59].

None of the above approaches take into account depthperception, image saliency, identification of the best rota-tion axis, identification of good pivot points for fixation, ordepth re-mapping to generate high-quality KDE that we have

used in our approach. Another significant departure in ourapproach has been the decoupling of the rendering camerafrom the acquisition camera. This has allowed us signifi-cantly greater freedom in using the standard angular cameramotion with substantially fewer visual artifacts as well asusing previously unexplored camera motions to achieve theKDE.

4 Overview of our approach

The input to our system is a pair of images that can generatean approximate depth map. Our approach also works withan image and a depth-map pair of the scene available froma camera coupled with a depth sensor such as the MicrosoftKinect.

We will refer to the cameras used in taking the initialimages as input cameras and the cameras used for generatingthe animation (to simulate the KDE) as rendering cameras.Our system is able to generate this animation through a seriesof steps as shown in Fig. 1. First, from the input stereo imagepair we compute the optical flow and a depth map. Afterthat, we compute parameters needed to generate a KDE bytaking into account depth, centrality, saliency, and opticalflow. Depth perceived by kinetic motion depends on the rel-ative velocity. Using the result of the user study (that allowsus to map velocities to perceived depth which is explainedin Sect. 6) and image saliency, we re-map the depth andcreate a depth mesh. This remapping reduces total motionwhile enhancing the depth perception. Finally, we visualize

Fig. 1 We first compute the optical flow and the depth map from theinput images (generally a stereo pair). We then generate a triangulateddepth mesh using the re-mapped depth map which is guided by humanperception.We also create an energymap of the image using depth, cen-

trality, and image saliency. By using the depthmesh and the energymap,we generate a high-quality animation to best experience the kinetic-depth effect. Our system has been informed by and validated throughuser studies

123

1360 S. Bista et al.

the depthmesh using any desirable rendering cameramotions(such as angular or conical-pendulum) to generate the anima-tion needed to experience the KDE. Details of our approachare described in the following sections.

5 Kinetic-depth effect parameters

WegenerateKDI by calculating proper values for pivot point,rotation axis (shown in Fig. 2), magnitude of angular rota-tion, and frequency. The pivot point is the look-at point ofthe rendering cameras as they move about the rotation axis.In this paper, we use two different types of camera motionand each type uses the rotation axis in a slightly differentway. We discuss this further in Sect. 9. The pivot pointis also a position in the image through which the rotationaxis passes. This results in minimal optical flow near theregion around the pivot point when the final animation iscreated.

To create an effective kinetic-depth experience, we need todetermine the magnitude of angular rotation and frequency.Wallach andO’Connell [53] used an angle of 42◦ at the rate of1.5 cycles/s. Epstein [13] used angles of 15◦ and higher in hisexperiment and reported that 15◦ is sufficient to perceive theKDE. His experiment used the projection of a shadow withno visible shading and texture. Since we are using typicalnatural stereo images with full texture, we have found thatrotation as small as 0.5◦ about the pivot point is sufficient.This is consistent with human vision, where an average intra-ocular distance is 6.25 cm, which gives about 4◦ separationat a fixation point 1 meter away. For objects farther awayseparation is much less. In our results, we perform rotationsbetween 0.5 to 2◦ around the pivot point. The frequency ofrotation used in our system is 2 cycles/s since this has been

Fig. 2 The pivot point and the rotation axis of the scene are shown inboth 3D space and the projection space

reported as the optimum temporal frequency by Nakayamaand Tyler [36], Caelli [3], and Rogers and Grams [44] .

Although we keep the rotation at the pivot point small,objects that are close or far might exhibit much higher move-ment, depending on the scene. According to Ujike et al. [50],30◦ to 60◦/s on each axis produces the highest motion sick-ness; roll produces more sickness than pitch and yaw. Takingthis into account, we keep the maximum rotation over theentire scene low and keep the change in the vertical axis to aminimum.

The positioning of the pivot point, frequency of rotation,magnitude of angular rotation, and the scene depth rangeall directly affect the velocity of the stimulus moving onthe screen. Gibson et al. [14] and Rogers and Grams [43]have shown that depth perceived by kinetic motion dependson the relative motion. Although some increase in relativemotion enhances the perception of depth, excess motioncauses motion sickness and reduces the ability to smoothlytrack objects. Robinson et al. [42] showed that while track-ing objects with velocity of 5◦/s , the human eye took about0.22 s to reach the maximum velocity and about 0.48 s toreach the target steady-state velocity. In our experiment, thefrequency of rotation is 2 or more cycles and, therefore,to maximize KDE and at the same time reduce unwantedmotion-based artifacts, we keep motion below 5◦/s. Usingdepth re-mapping based on results obtained in Sect. 6, weminimize motion artifacts while maximizing the perceptionof depth.

Severalmore variables are calculatedwhenweoptimizeKDI.They are described below.

Relative distance: The relative distance between two ver-tices v and w is defined by Rd(v,w) = ‖vd − wd‖2.Velocity: The velocity of a vertex v is computed by tak-ing camera parameters, rendering motion, and viewing setupinto consideration. So, we first compute positions of v in thescreen space at times t0 and t1. We call them vt0 and vt1 . Todo this, we use camera motion parameters to generate a KDEand project them on computer screen. Next, the velocity of v

in screen space is computed by equation Vv = vt1−vt0t1−t0

. Then

the velocity Vv is converted into angular velocity expressed

using view angle by equation Vv = Vv(2∗tan−1(pixelSize∗.5)

viewing Distance

).

Relative velocity: Relative velocity between two vertices vand w is defined by Rv(v,w) = Vv − Vw.

Disocclusion threshold: When viewing a depth mesh witha moving camera, regions that were occluded from the inputcamera used to create the depth mesh could become visi-ble. As we draw a triangulated depth mesh, these deoccludedregions are filled by the stretched triangles (also known asrubber sheets) that span the depth discontinuity from theedge of the foreground object to the background [7,33]. To

123

Kinetic depth images: flexible generation... 1361

minimize the disocclusion artifacts, the rendering cameraplacement and movement to experience the kinetic-deptheffect should be constrained. However, even after perform-ing the optimizations some disocclusion artifacts remaindue to the structure of the scene. To quantify amount ofperceptible disocclusion when KDI is generated, we cal-culate disocclusion estimates between two vertices v andw defined by Oocc(v,w) = ∥∥vt − wt

∥∥∞ ∗ Clab(vcol, wcol)

where the maximum screen space difference over the entireKDE animation is multiplied by the function Clab whichcomputes color difference in CIE LAB color space. To deter-mine disocclusion of the entire mesh OMeshOcc, we takek largest Oocc and compute average as given by equationOMeshOcc = 1

k

∑k Oocc.

6 Experiment

We performed an experiment to determine the relation-ship between velocities, positioning, and depth perception.Numerous studies have been carried out to determine whatcauses motion-based depth perception and its implicationson segmentation, depth ordering, and depth magnitude esti-mation [12,14,26,36,37,43,44,52,56]. Most of them studymotion in isolation and focus only on relative velocities.Since we generate a KDE based on stereo images, for usthe relationships between depth perception, spatial perspec-tive, and the positioning of the pivot point in the scene isvery important. All of these factors influence our depth per-ception generated byKDE. To our knowledge, there has beenno attempt thus far in examining the aggregate relationshipamong these factors and KDE. Specifically, we examine thecombined effect on depth perception from the following fac-tors: (a) the background and foreground of an object placedat the pivot point move in opposite directions, while for anobject distant from the pivot point they move in the samedirection; (b) the average velocities of the objects placed atthe pivot point are much lower than for objects that are dis-tant from the pivot point; (c) the relative velocity betweenthe foreground and the background of an object decreasesconsiderably when receding from both the view point andthe pivot point due to the perspective effect, and finally (d)the pivot point has no motion.

We recruited volunteers from our campus and the localcommunity for the experiment. All volunteers had normal orcorrected-to-normal vision and were able to perceive KDE.The experiment was performed by following the universityIRB guidelines and the participants were compensated nomi-nally for their time and effort. The experiment was conductedin a brightly lit lab using a Dell monitor 2407WFPH withpixel pitch 0.270 mm. The distance between the participant’seye and the monitor was around 0.75 m, which is within theOSHA recommended range [51]. We showed stimuli, which

Fig. 3 The image a shows the side view of the object being displayedin the experiment rendered with lighting for illustration purposes. Theimage b shows the stimulus shown to the participants. Without KDEmotion, the structure is seen as a flat collection of points

subtended around 20◦ ×20◦ visual angle, made of randomlygenerated dots with various depths. When motion was notpresent, the randomly generated dots were perceived as a flatfronto-parallel plane (Fig. 3b).

The experiment was done to study the relationshipbetween relative velocity, depth perception, and the posi-tioning of the pivot point. In this experiment, we renderedobjects with various depths using randomly generated dots.The objectswere composed of a planewith a box either goingin or coming out as shown in Fig. 3. The variation of depthbetween the front and the back of the object changes the rel-ative velocities between them. The objects were placed atvarious distances from the pivot point, which changes rota-tional velocity. The objects close to the pivot point experienceless velocity. The positioning of the pivot point, scene depth,and spatial perspective affects the relative velocity of pixelson the projected screen. For the study, the relative velocitybetween the foreground and the background was in the range0.02◦–2.5◦ visual angle/s. In this study, 11 volunteers partic-ipated.

Method: Each participant performed 40 trials with 3 sintervals between trials. There was no restriction on time.Participants were asked to estimate the size of the stimulususing a slider. The specific instruction to them was Use theslider to show us how much depth you perceive. The userswere then supposed tomove the slider from its origin to a dis-tance that visually corresponded to the depth that they wereperceiving.

Result:We found that the estimation of the perceived depthbetween subjects varied dramatically. This is consistent withthe previous study [12], where the researchers reported ahigh variation in perceived depth. However, when depth val-ues are normalized per subject, a distinct pattern emerged.We computed the average of the normalized depth for allsubjects for each distance from the pivot point, as shown inFig. 4. The participants perceived increased depth betweenrelative velocities of 0.2◦–1.25◦ visual angle/s; for higherrelative velocities, the perceived depth remained constant.

123

1362 S. Bista et al.

Fig. 4 The figure shows the mean and standard deviation of the nor-malized depth perceived by subjects. When the objects are placed atvarious distances from the pivot point, the perceived velocity of theobject changes. We show separate curves for each distance we testedand list the midpoint velocity. Between relative velocities of 0.2◦ to1.25◦ visual angle/s, participants perceived increased depth. For higherrelative velocities, the perceived depth remained constant

Thus, we consider 1.25 as the maximum desirable relativevelocity. This is taken into consideration while generatingthe depth mesh. We performed a two-way analysis of vari-ance (ANOVA) between the distance from the pivot pointand the relative velocity in the field of view (view angle).The results of ANOVA for the distance from the pivot pointare [(F(3, 400) = 3.16, P = 0.0248], the relative velocityare [(F(9, 400) = 108.94, P < 0.0001], and the interac-tions between them are [(F(27, 400) = 0.96, P = 0.5208].These values indicate that both the distance from the pivotpoint and the relative velocity affect the perceived depth, butthere is no evidence of an interaction between the two. Pleaselook at the accompanying video for a visual explanation ofthe results.

Perceptualmaps: Our experiment and previous studies haveshown that relative velocity is an important motion cue toexperience KDE [14,43,46]. By applying Gaussian filter tosmooth the results from this experiment, we compute a setof curves that are used to map depth perception to relativevelocity and vice versa. We generated two sets of curvesto approximate the results from the experiment as shown inFig. 4. The first set of curves maps the relative velocities tonormalized depth perception (RV2NDPmap) and the secondset maps the users’ normalized depth perception to the rela-tive velocities(NDP2RV map). The inverse of the RV2NDPmap is the NDP2RV map. The curves within each set showhow the relation between relative velocity and depth per-ception changes as the midpoint velocity is changed. Thismapping gives an empirical relationship between the nor-malized depth perception and the relative velocity.

7 Energy map computation

As the pivot point experiences minimal optical flow, it ispreferable to locate the pivot point at a salient region of thescene. If the pivot point salient region has text or a face,it becomes easier to read or identify it when the region isnot exhibiting high optical flow due to the rendering cameramovement. It is also preferable to have the pivot point in themiddle of the scene to minimize the optical flow at the sceneboundaries. If the pivot point is chosen at a scene boundary(as an extreme example), the opposite end of the scene willexhibit excessive motion that can create visual discomfortand can also give rise to significant disocclusion artifacts.

To quantify and incorporate such considerations,we deter-mine the placement of the pivot point by computing an energymap. We seek a global minimum on the energy map to deter-mine the kinetic-depth parameters:

E(x, y) = Ed(x, y) + Es(x, y) + Er(x, y), (1)

where (x, y) refers to the position of the pivot point in theoriginal image (projection space) and Er(x, y), Ed(x, y) andEs(x, y) are the radial, depth, and saliency energy functions,respectively. Each energy function component is further dis-cussed below.

Depth energy: We use the depth energy to express a prefer-ence for positioning the pivot point on regions that are closeto the middle of the depth map. This minimizes the visibilityof the occluded regions during the rendering camera move-ments and also lowers the amount of motion when the finalanimation is generated. We obtain the depth map of a givenimage set by calculating its optical flowusing Sun et al.’s [48]algorithm. Details about depth map calculation are describedin Sect. 8. After calculating the depth map we can calculatethe depth energy as Ed(x, y) = ‖Pd(x, y) − Dm‖2 , wherePd(x, y) refers to the depth value of the pixel at (x, y) andDm is the median depth of the scene. The second image ofFig. 5 shows the depth map used in Pd(x, y).

Fig. 5 Energy components. Here we show a the original image, b thedepth map, c the saliency component, d the radial component, and e thefinal computed energymap. In thesemaps, yellow regions are associatedwith lower energy. We choose the pixel with the lowest energy to be thepivot point

123

Kinetic depth images: flexible generation... 1363

Saliency energy: As mentioned earlier, we desire to posi-tion the pivot point on a salient region of the scene. Thesesalient regions are different in color, orientation, and inten-sity from their neighbors and can be found using the imagesaliency algorithm proposed by Itti et al. [21]. The saliencymap represents high-saliency regionswith high values.Whencalculating the saliency energy of the pivot point, we invertthe saliency values so that the most salient regions are repre-sented with the lowest values. Our equation for calculatingthe saliency energy is Es(x, y) = [1 − Ps(x, y)], where Psrefers to the saliency value of the pixel. The third image inFig. 5 illustrates the saliency map used in Ps(x, y).

Radial energy: To express a preference for a centralizedpivot point, we use a radial energy function as one of theenergy components. Essentially, the closer the point is tothe center, the less radial energy is associated with it. Theradial energy is calculated as Er(x, y) = Pr(x, y), where Prrefers to the radial value of the pixel defined by the Euclideandistance between the point and the image center. The fourthimage in Fig. 5 shows an example of the radial map.

The radial component Pr(x, y) depends upon the dimen-sions of the image, but the saliency component Ps(x, y) andthe depth component Pd(x, y) depend upon the scene. Totake these factors into account, we can add weights whilecalculating the energy function. Assigning a higher saliencyweight will increase the importance of the salient regions,whereas higher depth weight and radial weight will givegreater priority to the image center and lower the totaloptical flow between frames. Figure 5 shows an exampleof energy map calculation. We compute the energy of allthe pixels and find the position (x, y) that has the lowestenergy. Then the pivot point is defined by the coordinate[(x, y, Pd(x, y)].

8 Mesh generation

We generate the depth mesh by first approximating the scenedepth, followedbyanoptimized compression forKDE.Then,we perform re-mapping of the depth mesh by taking percep-tion into account. Finally, we enhance the depth mesh so thatthe motion is constrained to a desired range and disocclusionartifacts are minimized.

8.1 Scene depth approximation

Generally the number of input images or photographs issmaller than the desired number of views. Also, the para-meters used by the rendering cameras are often differentfrom the input cameras. Due to these reasons, a numberof additional intermediate frames need to be generated.There are numerous ways of generating intermediate frames,

such as basic interpolation between input images, flow-fieldinterpolation [61], image-based rendering [2,6], and struc-ture approximation [1,5,15,17,31,34,38,45,58,60] to createdepth mesh.

In our approach, we use a depth-image-based rendering togenerate high-quality intermediate frames needed for achiev-ing the KDE. This is computationally efficient and allowssufficient flexibility in the choice of the rendering cameraparameters. For every pair of images, we first calculate thedepth map. Although there are numerous methods to com-pute the depth map, we decided to approximate depth basedon the optical flow between the input image pair by usingthe inverse relation defined as d = f t

o , where d is thedepth, f is the focal length, t is the distance between thecamera positions, and o is the optical flow magnitude of thepixel. Since we do not know the camera parameters f and t ,we recover depth up to a projective transformation. Opticalflow between images is a vector field. Objects at the zeroplane will have zero optical flow. Objects that are in frontand behind the zero plane will have optical flow vectors fac-ing in the opposite directions. Taking either the maximumor minimum optical flow and adding it to the entire vec-tor field will shift the zero plane to either the closest or thefurthest depth. We then convert optical flow map to depthmap.

8.2 Optimized depth compression

Depth range on a raw depth map is usually very high andcontains a lot of artifacts. Directly using a raw depth mapwill cause excessive motion that is visually disconcerting.One naïve solution to this problem is to perform simple scal-ing of the depth map to fit a certain range. However, thiswill compress both important and unimportant regions ofthe scene uniformly. Extensive amount of research has beenconducted in visual saliency and perceptual enhancements[9,10,16,20,21,23,24,28,29,57]; however, use of motionmakes our situation unique. We want the salient regions ofthe scene to occupy a larger share of the depth range whilecompressing non-salient regions and artifacts. An analogousproblem arises in stereo viewing and relief mapping, wheredisparity or depth compression is needed [22,27,30,40,54].InKDI, compression requires global consistency of the scenedepth as the depth mesh is viewed from different angles; oth-erwise, the depth inconsistencies will be perceived easily.The disparity histogram and saliency map have been used byLang et al. [27] to compress disparity for stereo viewing. Inthe slightly different problem of video retargeting, manualconstraints have been used to enforce feature preservationwhile compressing the frames [25].

We perform depth compression based on the imagesaliency and use cues from carefully chosen edges to enforcefeature preservation. Our approach for compressing depth is

123

1364 S. Bista et al.

similar to that of Lang et al. [27]. We would like to accountfor saliency in compressing depth. To compress depth, wefirst divide the depth range of themesh evenly into k intervals{r0, r1, ....rk−1}.Wewould like to have a greater compressionfor depth intervals that are largely empty or have low-saliencevertices. To achieve this, we build a histogram over the depthintervals, rx in which we use vertex counts weighted by theirrespective saliencies (each vertex’s saliency is in the range0 to 1). Next, we compute a compressed value sx for theinterval rx as follows:

sx = min

(hx

g ∗ max(h0, h1, ....hk − 1), 1.0

), (2)

where sx is the size of the interval x after using saliencycompression, and g is a constant which gives extra rigidityto salient depth intervals. In practice, we found that g =0.4 gives a proper balance between compression and rigidityof depth intervals. This non-linear mapping compresses theintervals that are less salient.

Only using saliency to compress the depth map willcause features, such as lines and curves, to change. This isbecause depth intervals are compressed non-linearly. Someconstraints have to be placed to preserve features. We useinformation of carefully chosen edges to enforce loose fea-ture preservation. We first calculate edges in the image usingCanny edge detection [4]. If needed,more sophisticated edgedetectors canbe easily integrated.Wefilter edges that are veryshort in length to remove noise. After that, we perform addi-tional filtering to remove edges that lie at the border regionof objects at various depths. This is done by removing edgeswith very high gradient on their depth values. This step isimportant because the depthmap contains errors/artifacts andthe depth approximation close to the depth boundaries is lessreliable. Edges that are left after filtering will be longer andwill have more reliable depth values. These edges are usedfor feature preservation. We find the depth interval associ-atedwith each of these filtered edges and uniformly distribute

depth among them as s∗x =

∑ jn=i snj−i , where s∗

x is the new sizeof the depth interval x after using saliency compression withfeature preservation; i and j are the depth interval associ-ated with the two endpoints of a filtered edge and x ∈ [i, j].Finally, using the compressed depth, we create compresseddepth mesh MComp.

8.3 Perceptual re-mapping

Mesh re-mapping: We generate the re-mapped depth meshby taking depth perception into account. Since we would likethe viewers to perceive depth that is close to the 3D structurewe present, we use the relative depth (separation) computedfrom the compressed depth mesh MComp as an input for theNDP2RV map (from Sect. 6) to generate a relative velocity

map needed to perceive the desired depth. This step is donefor each vertex v in MComp. For each v, let Vn be the set ofits four connected neighbors such that Vn ⊆ MComp. We cre-ated four vertex-neighbor pairs (v,w) where w is in the setVn. Then for each pair (v,w), we compute the relative depthRd(v,w) between them. The relative depth is scaled so that itis within the maximum depth range allowed in the scene. Atfirst, maximum depth range is initialized to the depth rangethat results in a relative velocity that is equal to the maxi-mum desirable relative velocity which is explained in Sect. 6.The scaled depth is then used as an input for the NDP2RVmap to get the relative velocity Vp(v,w) between the pair.We call Vp(v,w) a perceptual relative velocity because it isbased on the perceptual relationships explained earlier. TheVp(v,w) between a vertex pair is necessary to perceive therelative depth Rd(v,w) between them. In other words, toperceive a depth that is equal to Rd(v,w) between a vertexpair, we need the relative velocity between them to be equalto Vp(v,w).

Using the perceptual relative velocities between vertices,we can compute their separation (we call this perceptual sepa-ration Sp(v,w)). Wemove one of the vertices in (v,w) alongthe line of the view point and find separation Sp(v,w) thatresults in the desired Vp(v,w). The relative velocity betweena pair of vertices is computed by projecting each vertex onthe display screen using a standard projection matrix, thencalculating the instantaneous velocity (described in Sect. 5)for each vertex, and finally finding the difference betweenthe velocities. This process is accelerated by a binary searchalgorithm. Since this step is performed by only using localdata on the distances between neighbors (v,w), an extrastep to make a globally consistent mesh is required. Thisis done by minimizing a 1D problem. We first discretizedepth of the scene into 255 bins. Then for each (v,w), wefind associated discretized depths dv and dw based on theirdepth in MComp. We then add a link between dv and dwthat specifies the Sp(v,w). This process can be pictured as aspring mesh where the links added will act like a spring andthe discretized depths are the locations where the springsare attached. Since the Sp(v,w) is locally consistent, somelinks will try to contract the distance between the discretizeddepths while some will try to expand them. We define *d(v,w) =| dv − dw |. For all (v,w), we find d(v,w)

that minimizes∑∥∥d(v,w) − Sp(v,w)

∥∥2 to perceptually

re-map the depth mesh MP to make it more globally con-sistent.

8.4 Depth mesh enhancement

In Sects. 8.2 and 8.3,we have carried out local depth enhance-ment using local per-pixel neighborhoods. However, this cancause the range of the motion for the entire image to be toolow or high. In this section we carry out a global optimization

123

Kinetic depth images: flexible generation... 1365

for a more pleasant viewing experience. Depending on therendering parameters and the depth range of the scene, depthmesh can be compacted or expanded to allow enhanced depthperception. Also, depth has to be adjusted to reduce disocclu-sion artifacts. In Sect. 6 we mention that when using KDE,we have a limited perception of depth, which is highly depen-dent on the relative velocity. The relative velocity of objectsdepends on the relative distance between them, the placementof the pivot point, and the camera movement. However, thereis a tradeoff between the amount of motion in the scene dueto relative velocity and perception of depth. The results ofSect. 6 give us the maximum desirable relative velocity, tak-ing into account the tradeoff. Using the maximum desirablerelative velocity, perceptual depthmesh, and placement of thepivot, we find the maximum depth range Dmax allowed in thescene. In otherwords, givenperceptual depthmesh andplace-ment of the pivot, we find depth range Dmax which results in arelative velocity equivalent to themaximumdesirable relativevelocity between the front and back extremes of the mesh.This is done by searching for the depth range that resultsin the desirable relative velocity. This process is acceleratedby a binary search algorithm. Then we define the minimumdepth range allowed in the scene Dmin = Dmax ∗ 0.5.

Once we find the depth range allowed, we focus our atten-tion on finding the depth range that minimizes disocclusionartifacts. In between vertices, we estimate disocclusion bycomputing relative separation in pixel units between neigh-boring vertices of the depthmeshwhen the cameramovementneeded forKDI is performed.However, when colors betweenthe vertices are the same, disocclusion is not visible and wecan ignore disocclusion between these vertices. To approxi-mate the disocclusion of an entire scene OMeshOcc, we takethe mean of k-highest disocclusion estimates between ver-tices whose color is perceptually different as described inSect 5. When the depth range is small, disocclusion artifactsas well as the perception of depth is reduced. So, to find aproper balance, we compute the depth range by

DFinal =

⎧⎪⎨⎪⎩

DMin, if (DOpt ≤ DMin)

DMax, if (DOpt ≥ DMax)

DOpt, otherwise,

(3)

where DOpt is the depth range that has k-highest disocclusionestimates that are less than a user-specified threshold (herewe use threshold value of 2). If the difference between DFinal

and the depth range ofMP is small, we simply scale the depthrange of MP to match DFinal. However, if the depth range islarge, we modify the maximum depth range allowed in thescene and start the perceptual re-mapping process again. InFig. 6, we show a comparison between the raw depth andthe depth used to generate final depth mesh after applying allthe optimizations stated here.

Fig. 6 a The original image, the image set b is associated with theraw depth, and the image set c shows the depth used to generate finaldepth mesh after depth compression and perceptual enhancements. Thedepths in b and c are normalized. In each set we are showing the depthvalues and the optical flow after the camera motion to depict the valueof depth remapping

Fig. 7 An illustration using input image (a) and optimized depth map(b) of the different types of camera movements : the angular motion (c)where the camera swivels on a plane perpendicular to the rotation axiswhile looking at the pivot point, and the conical pendulum motion (d)where the camera rotates along a conical surface while looking at thepivot point. Rotation axis is always vertical for the angular motion andis perpendicular to the view-plane for the conical pendulum motion

9 Rendering camera motion

We make use of the pivot point and the rotation axis calcu-lated earlier to compute the rendering camera motion in twoways: angular motion and conical pendulum motion.

Angular motion to experience the KDE involves rotatingthe camera on a plane perpendicular to the rotation axiswhile looking at the pivot point as illustrated in Fig. 7c. Theangular motion is close to most of the wiggle stereo imagesfound online. However, rather than just flipping between twoimages and only relying on the sequence of images that werecaptured by the camera, we generate the intermediate viewsby rotating the rendering camera positions along an arc sub-tending a fixed angle at the salient pivot position. Figure 8shows our results using the angular motion.

123

1366 S. Bista et al.

Fig. 8 Representation of the angular camera motion

Conical pendulum motion involves rotating the cameraalong a circle on a plane parallel to the view-plane, as thevector from the camera to the look-at pivot point traces outa cone. Figure 7d illustrates the conical pendulum motionperformed by the camera.

10 Rendering and interaction

The depth mesh, along with the camera motion describedearlier, is used for rendering the scene. The depth mesh iskept static while the rendering camera moves to generate theKDE. Since we compute the depth mesh, our method allowsus to add virtual objects or different layers into the sceneat user-specified depth locations. We can, therefore, use thesame camera parameters to render additional geometry at thedesired locations while also rendering the depth mesh. Thismakes the kinetic-depth motion consistent for both the depthmesh as well as the additionally-added geometry. Currently,we do not attempt to make lighting and shading seamlessand consistent between the virtual object and the scene, butit would be an interesting exercise to attempt to achieve it byestimating the lighting parameters or by using methods suchas Poisson image editing [39].

Although we compute the pivot point and the rotation axisautomatically as a default, our system also allows the userto change these features as desired. This allows the user tocustomize the output of the kinetic-depth movement accord-ing to their needs. As mentioned earlier, the area around thepivot point has less motion. If there is more than one regionthat has low energy in a scene, being able to move the pivotpoint to various locations is critical.

11 Results and discussion

We have implemented our system in C++ andMatlab. For allof our experiments, we use aWindows 7 64-bit machine withan Intel Core i5 2.67 GHz processor, an NVIDIA GeForce470 GTX GPU, and 8 GB of RAM. We calculate imagesaliency using the graph-based visual saliency algorithm ofHarel et al. [16] implemented in MatLab and use Sun em etal.’s [48] code for optical flow calculations. The renderingof KDI is done at interactive frame rates; however, the cre-

ation takes time. The optical flow and saliency approximationalgorithms takesmajority of the computation timeduring cre-ation. On average, one megapixel image takes about 4 min.However these pre-processing steps are not currently opti-mized for speed.

11.1 Subjective evaluation

In order to evaluate the perceptual quality of the generatedKDI images, we conducted a user studywith 11 subjects whoalso participated in the earlier user study. Each subject per-formed four different tests to compare between: (1) Wiggle3D and our proposed method, (2) the naïve method and ourproposed method, (3) Piku-Piku and our proposed method,and (4) angular and conical cameramotion.We presented tenexamples for each test selected randomly from a larger col-lection of examples. Each example showed two images sideby side. Subjects were asked which image they preferred bytaking into account perceived depth and motion. The specificquestion to the subjects wasDo you prefer the left image, theright image, or have no preference for either?. The subjectsdid not knowwhich imageswere generated usingourmethod.

Naïve vs. our method: The naïve method uses a scaleddepth mesh (the magnitude of raw depth range scaled to thesame range as our method) with the pivot point selected atthe middle of the scene. The same camera motion in ren-dering was used for both the naïve method and our method.Out of 110 examples, subjects selected the naïve method 21times, our method 83 times, and had no preference 6 times.In Fig. 9b, we show the user preference per subject. We per-formed ANOVA with [(F(1, 20) = 53.24; P < 0.0001]which shows that the difference is significant.

Wiggle 3D vs. our method: Generally, wiggle 3D imagesare created by an artist by manually aligning the most salient

Fig. 9 The results of the subjective evaluation. Here we show the aver-age user preference

123

Kinetic depth images: flexible generation... 1367

region of the images. To generate Wiggle 3D image, mostsalient regionof the images is calculatedusing image saliencyalgorithm proposed by Itti et al. [21]. Then the zero planeposition is selected to coincide with the most salient part ofthe scene usually located around the central region of thescene. Out of 110 examples, subjects selected Wiggle 3Dimages 17 times, our method 84 times, and had no preference9 times. In Fig. 9a, we show the user preference results persubject. We performed ANOVA to analyze the differencebetween user selection of Wiggle 3D and our method. Wegot [(F(1, 20) = 34.21; P < 0.0001], which indicates thatthe difference is significant.

Piku-Piku vs. our method: Here we compared our methodwith Piku-Piku images from Start3D. Out of 110 exam-ples, subjects selected Piku-Piku images 8 times, our method87 times, and had no preference 15 times. In Fig. 9c,we show the selection results according to each type ofcamera motion per subject. We performed ANOVA with(F(1, 20) = 55.04; P < 0.0001) which indicates that thedifference is significant.

Angular vs. conical motion: We have compared imagesgenerated by two camera motions to find which methodsubjects preferred. Out of 110 examples, subjects selectedangularmotion 51 times, conicalmotion 52 times, and had nopreference 7 times. In Fig. 9d, we show the selection resultsfor each type of camera motion per subject. We performedANOVA with [(F(1, 20) = 0.02; P = 0.8987], which indi-cates that the difference is not significant.

Discussion: We took a detailed look at the examples wherethe subjects chose other alternativemethods over ourmethod.In most of these examples, we found that the depth rangewas low and the salient objects were already in the middle ofthe scene. In these examples, all of these methods generatedcomparable output. Also, both our method and Piku-Pikuimages rely on the computed depth maps. Any error in depthmaps will affect the final output.

Wiggle3D and Piku-Piku images are highly dependent onthe input pair of stereo images to generate the final rendering.They either switch or generate intermediate frames betweenthe input images. Based on the angle of the input camera,the images may contain scaling or shearing. The perceptionof depth is reduced when noise, scaling, or shearing appearsin the animated images. This can be easily observed whenwatching the animation. However, these subtle artifacts aredifficult to show using the vector plot, so we would like torequest that the reviewers look at the accompanying video.

11.2 Limitations of our approach

Although the use of the KDE to help understand the three-dimensional structure of a scene is valuable, it also has somedisadvantages. First, adding motion could be visually dis-

tracting and has the potential to induce motion-sickness.Although this can be considerably reduced by making thecamera motion small and smooth and by depth re-mapping,it cannot be completely eliminated. Second, the depth per-ceived by the KDE is not the same as the depth perceivedby binocular disparity. In fact, Durgin et al. [12] have shownin their experiments that depth judgments based on binocu-lar disparity are more accurate compared to depth judgmentsbased on the KDE or motion parallax. However, binoculardisparity perceived using stereoscopic displays has its owndisadvantages such as the vergence and accommodation con-flicts, and visual fatigue [19,47]. To reduce the visual fatiguein stereo displays, various techniques that compress the depthare also used [19], and they too are likely to reduce the accu-racy of the depth judgments. Third, our algorithm relies oncomputing approximated depth maps from a set of stereoimages. When the depth map calculation has a significanterror, the quality of our animation could also suffer. Thus,both ways of looking at 3D scenes have their own advan-tages and disadvantages. However, the use of the KDE issimpler and does not require any special devices.

12 Conclusions and future Work

Wehavepresented amethod to automatically create smoothlyanimated images necessary to experience the KDE. Given astereo image pair or an image-depth map pair as input, wehave presented an approach to automatically generate anima-tion that exhibits the KDE. Our approach allows decouplingof the input cameras from the rendering cameras to giveus greater flexibility in defining multiple rendering cameramotions.We have used two different ways of performing ren-dering cameramotions (angular and conic pendulummotion)to view the resulting scene that minimize visual artifacts bytaking into account depth perception, image saliency, occlu-sions, depth differences, and user-specified pivot points. Wehave reported the results of user studies that examine the rela-tionship between depth perception, relative velocity, spatialperspective effect, and positioning of the pivot point whengeneratingKDI.Wehave presented a novel depth re-mappingmethod guided by perceptual relations based on the resultsof our user study. And finally, we have presented a subjec-tive evaluation of our method by comparing it against otherexisting alternatives on a wide range of images.

At present, we have optimized one variable at a timeto do depth enhancement. A single optimization across allthe pixels is likely to lead to superior results and would beworth exploring. Another future direction is to explore theusage of KDE on modern 3D games to enhance the per-ception of the 3D structures. KDI may very well becomea popular 3D previewing tool for scientific visualizationapplications where depth awareness is important (such asstereo microscopy) and for consumer-entertainment applica-

123

1368 S. Bista et al.

tions including technology-mediated social community sites.Another important group that could benefit from KDI are thepeople that have lost sight in one eye. Toyoura et al. [49]estimate that number to be around 300 million.

Acknowledgements This work has been supported in part by the NSFGrants 09-59979 and 14-29404, the State of Marylands MPower ini-tiative, and the NVIDIA CUDA Center of Excellence. Any opinions,findings, conclusions, or recommendations expressed in this article arethose of the authors and do not necessarily reflect the views of theresearch sponsors.

References

1. Agarwal, S., Snavely, N., Simon, I., Seitz, S.M., Szeliski, R.: Build-ing Rome in a day. In: International Conference on ComputerVision, pp. 72–79 (2009)

2. Buehler, C., Bosse, M., McMillan, L., Gortler, S., Cohen, M.:Unstructured lumigraph rendering. In: Proceedings of the 28thannual conference on Computer graphics and interactive tech-niques, pp. 425–432. ACM (2001)

3. Caelli, T.: On the perception of some geometric properties of rotat-ing three dimensional objects. Biol. Cybern. 33, 29–37 (1979)

4. Canny, J.: A computational approach to edge detection. IEEETrans. Pattern Anal. Mach. Intell 8, 679–698 (1986)

5. Chapiro, A., Heinzle, S., Aydın, T.O., Poulakos, S., Zwicker, M.,Smolic, A., Gross, M.: Optimizing stereo-to-multiview conversionfor autostereoscopic displays. In: Computer graphics forum, vol.33, pp. 63–72. Wiley, New York (2014)

6. Chaurasia, G., Duchêne, S., Sorkine-Hornung, O., Drettakis, G.:Depth synthesis and local warps for plausible image-based naviga-tion. ACM Trans. Graph. 32(3), 30:1–30:12 (2013)

7. Darsa, L., Costa, B., Varshney, A.: Navigating static environmentsusing image-space simplification and morphing. In: Proceedingsof the Symposium on Interactive 3D Graphics, pp. 25 – 34, 28–30April 1997

8. Davidson, C.: Piku-Piku. www.start3d.com (2012). Accessed Nov2012

9. Didyk, P., Ritschel, T., Eisemann, E., Myszkowski, K., Seidel,H.P.:Apparent stereo: the cornsweet illusion can enhance perceiveddepth. In: IS&T/SPIE electronic imaging, pp. 82,910N–82,910N.International Society for Optics and Photonics (2012)

10. Didyk, P., Ritschel, T., Eisemann, E., Myszkowski, K., Seidel,H.P., Matusik, W.: A luminance-contrast-aware disparity modeland applications. ACM Trans. Graph. (TOG) 31(6), 184 (2012)

11. Dosher, B.A., Landy, M.S., Sperling, G.: Kinetic depth effect andoptic flow-I. 3D shape from Fourier motion. Vis. Res. 29, 1789–1813 (1989)

12. Durgin, F.H., Proffitt, D.R., Olson, T.J., Reinke, K.S.: Comparingdepth from motion with depth from binocular disparity. J. Exp.Psychol.-Hum. Percept. Perform. 21, 679–699 (1995)

13. Epstein, W.: Perceptual invariance in the kinetic depth-effect. Am.J. Psychol. 78(2), 301–303 (1965)

14. Gibson, E.J., Gibson, J.J., Smith, O.W., Flock, H.: Motion parallaxas a determinant of perceived depth. J. Exp. Psychol. 58(1), 40–51(1959)

15. Gopi, M., Krishnan, S.: A fast and efficient projection-basedapproach for surface reconstruction. In: Computer graphics andimage Processing, 2002. Proceedings. XV Brazilian Symposiumon, pp. 179–186 (2002)

16. Harel, J., Koch, C., Perona, P.: Graph-based visual saliency. In:Advances in Neural Information Processing Systems 19, pp. 545–552. MIT Press, Cambridge (2007)

17. Hartley, R., Zisserman, A.: Multiple view geometry in computervision. Cambridge University Press (2003)

18. Heineman, J.: Stereogranimator. http://stereo.nypl.org/ (2012).Accessed Nov 2012

19. Hoffman, D.M., Girshick, A.R., Akeley, K., Banks, M.S.:Vergence-accommodation conflicts hinder visual performance andcause visual fatigue. J. Vis. 8, 33 (2008)

20. Ip, C.Y., Varshney, A.: Saliency-assisted navigation of very largelandscape images. Vis Comput Graph. IEEE Trans. 17(12), 1737–1746 (2011)

21. Itti, L., Koch, C., Niebur, E.: A model of saliency-based visualattention for rapid scene analysis. PAMI, IEEE Trans. 20(11),1254–1259 (1998)

22. Kellnhofer, P., Ritschel, T.,Myszkowski, K., Seidel, H.P.: Optimiz-ing disparity for motion in depth. In: Computer Graphics Forum,vol. 32, pp. 143–152. Wiley, New york (2013)

23. Kim, Y., Varshney, A.: Saliency-guided enhancement for volumevisualization. IEEE Trans. Vis. Comp. Graph. 12(5), 925–932(2006)

24. Kim, Y., Varshney, A., Jacobs, D.W., Guimbretière, F.: Meshsaliency and human eye fixations. ACMTrans. Appl. Percept. 7(2),1–13 (2010). doi:10.1145/1670671.1670676

25. Krähenbühl, P., Lang, M., Hornung, A., Gross, M.: A system forretargeting of streaming video. ACM Trans. Graph. 28(5), 126:1–126:10 (2009)

26. Landy,M.S., Dosher, B.A., Sperling, G., Perkins,M.E.: The kineticdepth effect and optic flow-II. First- and second-order motion.Vision Research 31(5), 859–876 (1991)

27. Lang, M., Hornung, A., Wang, O., Poulakos, S., Smolic, A., Gross,M.H.: Nonlinear disparity mapping for stereoscopic 3D. ACMTransactions on Graphics (TOG) 29(4) (2010)

28. Lee, C.H., Kim, Y., Varshney, A.: Saliency-guided lighting. IEICETrans. Inf. Syst. E92–D(2), 369–373 (2009)

29. Lee, C.H., Varshney, A., Jacobs, D.: Mesh saliency. ACM Trans.Graph. (Proc. SIGGRAPH 2005) 24(3), 659–666 (2005)

30. Lee, S., Kim, Y., Lee, J., Kim, K., Lee, K., Noh, J.: Depth manipu-lation using disparity histogram analysis for stereoscopic 3d. Vis.Comp. 30(4), 455–465 (2014). doi:10.1007/s00371-013-0868-3

31. Liu, W., Wu, Y., Guo, F., Hu, Z.: An efficient approach for 2D to3D video conversion based on structure from motion. Vis. Comp.31(1), 55–68 (2015). doi:10.1007/s00371-013-0904-3

32. Lytro: Perspective shift. www.lytro.com/camera#perspective_shift(2013). Accessed Dec 2013

33. Mark,W.R.,McMillan, L., Bishop,G.: Post-rendering 3Dwarping.In: Proceedings of the 1997 symposium on Interactive 3D graphics,I3D ’97, pp. 7–10. ACM, New York, USA (1997)

34. Michailidis, G.T., Pajarola, R., Andreadis, I.: High Performancestereo system for dense 3-D reconstruction. Circuits and systemvideo for technology. IEEE Trans. 24(6), 929–941 (2014)

35. Nagata, S.: How to reinforce perception of depth in single two-dimensional pictures. In: Stephen, R.E. (ed.) Pictorial communi-cation in virtual and real environments, pp. 527–545. Taylor &Francis, Bristol (1991)

36. Nakayama, K., Tyler, C.W.: Psychophysical isolation of movementsensitivity by removal of familiar position cues. Vis. Res. 21, 427–433 (1981)

37. Ono, M.E., Rivest, J., Ono, H.: Depth perception as a func-tion of motion parallax and absolute-distance information. J Exp.Psychol.-Hum. Percept. Perform. 12, 331–337 (1986)

38. Patro, R., Ip, C.Y., Bista, S.,Varshney,A.: Social snapshot: a systemfor temporally coupled social photography. IEEE Comput. Graph.Appl. 31, 74–84 (2011)

39. Pérez, P., Gangnet, M., Blake, A.: Poisson image editing. ACMTrans. Graph. 22, 313–318 (2003)

40. Policarpo, F.,Oliveira,M.M.,Comba, J.L.D.: Real-time reliefmap-ping on arbitrary polygonal surfaces. In: Proceedings of the 2005

123

Kinetic depth images: flexible generation... 1369

Symposium on Interactive 3D Graphics and Games. I3D ’05, pp.155–162. ACM, New York, USA (2005)

41. Proffitt, D.R., Rock, I., Hecht, H., Schubert, J.: Stereokinetic effectand its relation to the kinetic depth effect. J. Exp. Psychol.-Hum.Percept. Perform. 18, 3–21 (1992)

42. Robinson, D., Gordon, J., Gordon, S.: A model of the smooth pur-suit eye movement system. Biol. Cybern. 55, 43–57 (1986)

43. Rogers, B., Graham, M.: Motion parallax as an independent cuefor depth perception. Perception 8, 125–134 (1979)

44. Rogers, B., Graham, M.: Similarities between motion parallax andstereopsis in human depth perception.Vis. Res. 22, 261–270 (1982)

45. Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: Exploringphoto collections in 3D. In: SIGGRAPH Conference Proceedings,pp. 835–846. ACM Press, New York, USA (2006)

46. Sperling, G., Landy, M.S., Dosher, B.A., Perkins, M.E.: Kineticdepth effect and identification of shape. J. Exp. Psychol.-Hum.Percept. Perform. 15, 826–840 (1989)

47. Stereographics: TheStereographicsDeveloper’sHandbook -Back-ground onCreating Images forCrystalEyes andSimulEyes. Stereo-graphics Corporation. http://www.reald-corporate.com/scientific/downloads/handbook.pdf (1997). Accessed Dec 2013

48. Sun, D., Roth, S., Black, M.: Secrets of optical flow estimationand their principles. In: Computer vision and pattern recognition(CVPR), 2010 IEEE Conference on, pp. 2432 –2439 (2010)

49. Toyoura, M., Kashiwagi, K., Sugiura, A., Mao, X.: Mono-glass forproviding distance information for people losing sight in one eye.In: Proceedings of the 11thACMSIGGRAPH international confer-ence on virtual-reality continuum and its applications in Industry,pp. 39–42. ACM (2012)

50. Ujike, H., Yokoi, T., Saida, S.: Effects of virtual body motion onvisually-induced motion sickness. In: Annual International Con-ference of the IEEE Engineering in Medicine and Biology Society(2004)

51. US. Dept. of Labor: Occuptional saftey and health administartion.www.osha.gov/SLTC/etools/computerworkstations/components_monitors.html (2013). Accessed Dec 2013

52. Vishwanath, D., Hibbard, P.B.: Seeing in 3-D with just one eye:stereopsis without binocular vision. Psychol. Sci. 24(9), 1673–1685 (2013)

53. Wallach, H., O’Connell, D.N.: The kinetic depth effect. J. Exp.Psychol. 45, 205–217 (1953)

54. Weyrich, T., Deng, J., Barnes, C., Rusinkiewicz, S., Finkelstein,A.: Digital bas-relief from 3D scenes. ACM Trans. Graph. 26(3),32 (2007)

55. Wilson, M.: Shooting challenge: Wiggle 3D. www.gizmodo.com/5895289/shooting-challenge-wiggle-3d (2012). Accessed Sept2012

56. Yoonessi, A., Baker, C.: Contribution of motion parallax to depthordering, depthmagnitude and segmentation. J.Vis. 10, 1194–1194(2011)

57. Yoshida, K., Takahashi, S., Ono, H., Fujishiro, I., Okada, M.:Perceptually-guided design of nonperspectives through pictorialdepth cues. In: Computer graphics, imaging and visualization(CGIV), 2010 Seventh International Conference on, pp. 173–178.IEEE (2010)

58. Zhang, C., Li, Z., Cheng, Y., Cai, R., Chao, H., Rui, Y.: Mesh-stereo: A global stereo model with mesh alignment regularizationfor view interpolation. In: International Conference on ComputerVision (2015)

59. Zheng, K.C., Colburn, R.A., Agarwala, A., Agrawala, M., Salesin,D., Curless, B.L., Cohen, M.F.: Parallax photography: creating 3Dcinematic effects from stills. In: Proceedings of Graphics Interface,pp. 111–118. Canadian Information Processing Society (2009)

60. Zhu, C., Leow, W.: Textured mesh surface reconstruction of largebuildings with multi-view stereo. Vis. Comput. 29(6–8), 609–615(2013). doi:10.1007/s00371-013-0827-z

61. Zitnick, C.L., Jojic, N., Kang, S.B.: Consistent segmentation foroptical flow estimation. Int. Conf. Comput. Vis. 2, 1308–1315(2005)

Sujal Bista is a graphicsresearcher and a game developer.His research interests includecomputer graphics, virtual andaugmented reality, scientificvisu-alization, artificial intelligence,and human-motion understand-ing. Bista did his Ph.D. and post-doctoral research at the Univer-sity of Maryland, College Park.

Ícaro Lins Leitão da Cunhais an assistant professor at theUniversidade Federal Rural dePernambuco. His research inter-ests include computer graph-ics, virtual reality and computa-tional mathematics. Cunha hasreceived his bachelor’s degreein computer science from theUniversidade Federal da Paraíba,an M.S. degree in computationalmathematics from the Universi-dade de São Paulo and a Ph.D.in computer engineering fromthe Universidade Federal do RioGrande do Norte.

AmitabhVarshney is the Direc-tor of the Institute for AdvancedComputer Studies (UMIACS),Professor of Computer Scienceat the University of Mary-land at College Park, and Co-Director of theCenter forHealth-related Informatics andBioimag-ing. Varshney’s research focusis on exploring the applicationsof high-performance computingand visualization in engineering,science, and medicine. He hasworked on a number of researchareas including visual saliency,

summarization of large visual datasets, and visual computing for bigdata. He is currently exploring general-purpose high-performance par-allel computing using clusters of CPUs and graphics processing units(GPUs). He has served in various roles in the IEEE Visualization andGraphics Technical Committee, including as its Chair, 2008–2012. Hereceived the IEEEVisualization Technical AchievementAward in 2004.He is a Fellow of IEEE.

123