A TOOLCHAIN FORCAPTURING AND RENDERING STEREO AND MULTI-VIEW DATASETS

Felix Klose, Christian Lipski, Kai Ruhl, Benjamin Meyer, Marcus Magnor

Institut fur Computergraphik, TU Braunschweig

ABSTRACT

We present our toolchain of free-viewpoint video and dynamicscene reconstruction for video and stereoscopic content cre-ation. Our tools take video data from set of sparse unsynchro-nized cameras and give great freedom during the post produc-tion. From input data we can either generate new viewpointsemploying the Virtual Video Camera a purely image basedsystem or generate 3D scene models. The approaches areexplained and guidelines for weighing their specific advan-tages and disadvantages in respect to a concrete applicationare given.

Index Terms— Stereoscopic 3D, Multi-View Stereo, Un-synchronized Cameras.

1. INTRODUCTION

Handling stereoscopic or multi-view material requires a spe-cialized set of algorithms. In order to capture, process and re-render such image sequences, typical challenges arise. Oneof which is the camera calibration, others are reconstructinggeometry or synthesizing novel viewpoints. A special case ofnovel viewpoints synthesis is the generation of a second eyeview for stereoscopic 3D (S3D) rendering.

The relative placement of multiple images or cameras isof special importance for S3D content generation. In a stereocamera rig the spatial distance between the camera viewpointsis the baseline and one of the most important parameters forthe viewers 3D experience. Additionally the temporal syn-chronization of stereo-rig cameras is mandatory to avoid view-ing discomfort caused by image content mismatches and ver-tical disparities.

Therefore camera calibration does not only require recov-ering the spatial camera parameters, but also the temporal off-set. This becomes even more an issue, when using more thantwo cameras or a recorder setup with heterogeneous framer-ates.

When recording with a stereo camera rig, the camera base-line is fixed at the recording time. The possibility to allowmodifications to the baseline or the overall 3D impression is

Funding by the European Research Council ERC under contract +No.256941 “Reality CG” and by the German Science Foundation DFG MA2555/1-3 +is gratefully acknowledged.

very desirable and an active research area. The same holdstrue for the temporal domain. Resampling a video in time canbe used to achieve slow motion effects or upsample footage,that is recorded at slow framerates.

Modifying S3D content on an even finer scale can en-able non-linear depth remapping to keep the 3D effects in theviewers comfort space. To achieve this per pixel remappingcapabilities, a scene depth reconstruction is necessary. Formulti-view setups, more information is available that can beexploited for reconstruction purposes, but the hardware ef-fort required to gen-lock and calibrate cameras greatly in-creases. With a given 3D scene model depth based render-ing enables re-rendering different or adaptive baselines froma single view.

For scene analysis or special effect purposes, it can beimportant to recover the motion from the recorded material.

We show a set of tools and algorithms that tackle thoseproblems, and create a great degree of freedom during thepost-production.

2. RELATED WORK

There exists a great deal of research for 3D reconstruction,rendering and production techniques. It would be far beyondthe scope of this paper to give a comprehensive overview.However only a few solutions focus on stereo- and multi-viewwork flows. Some groups have developed similar projects wewould like to highlight here:

The University of Surrey in cooperation with the BBC de-veloped iview [6]. Their multi-view free viewpoint system isfocused on performance and sport events. It is able to ren-der novel viewpoints such as first person perspective in sportsevents. The system is based on estimating a 3D reconstructionfrom multiple broadcast camera views and render the texturedgeometries.

Recently Disney Research [17] presented their tools fordisparity-aware stereo 3D production. They show for the stereo-scopic case what can be accomplished in the fields of non-linear depth remapping. Employing image-based warping tech-niques they show how to regain production freedom to in-crease the viewer comfort when watching S3D content.

For individual aspects of producing and editing stereo-scopic 3D content an increasing number of commercial tools

exist. The Foundry have a stereoscopic editing suite calledOcula [18] based on their compositing framework NUKE,which also allows adaptive image warping to change the 3Dimpression of video material.

This paper first gives an overview of our tools in Sec. 3.Then the individual aspects are presented in Sec. 4, Sec. 5and Sec. 6. The applications section Sec. 7 focuses on advan-tages and disadvantages of different tool choices and Sec. 8concludes the paper.

3. PRODUCTION WORKFLOW OVERVIEW

Fundamental principle of our work flow is to impose as fewrestrictions as possible on the recording modalities while pre-serving the greatest possible artistic freedom during post pro-duction. To make multi-view recording feasible, we focus onfootage recorded with multiple consumer grade cameras thatare not synchronized.

After recording, the processing of input material typicallybegins with spatially calibrating the cameras by determiningtheir parameters in a common 3D space. Recent structure-from-motion algorithms for unordered image collections [5]solve this problem robustly.

Following the spatial registration a temporal alignment ofthe camera feeds is performed. To determine the subframeaccurate offsets we use an algorithm described in Sec. 4 thatis based on the tracking of 2D image features.

With the fully calibrated input material our workflow of-fers two options. Our Free-Viewpoint rendering system (Sec. 5)is capable of rendering new in between views in space andtime. It thereby returns a great deal of freedom to the con-tent editor. The camera viewpoint including the temporal di-mension can be varied, S3D content can be obtained eitherrendering multiple views, or generating depth maps and usedepth-based rendering to create S3D. The latter also allowsfor modification of the depth impression. Being entirely im-age based, no explicit 3D scene model is necessary to do so.

The second option is to start a multi-view reconstructionprocess that recovers a dynamic scene model (Sec. 6). Theresulting model is patch based and each has an associated po-sition, normal and linear motion vector for each surface patch.

Both option have their own merits and applications, whichare discussed in section 7.

4. SUBFRAME TEMPORAL REGISTRATION

Our setup of unsynchronized cameras, while ideally suited tocasual capture, requires us to estimate the temporal alignmentin post production. Preferably, the alignment is performedwith subframe accuracy.

We divide our method into two steps: In the first, we trackfeature points [16] in two cameras over several frames andmatch their trajectories using extremal point and binary re-duction [3] and string matching [14]. This results in frame-

(a)

(b)

Fig. 1. Frame-accurate temporal alignment. The movementof e.g. a bouncing ball is tracked, simplified by taking itsextremal points (a) and reduction to binary codes (b). Finally,the binary codes are string matched.

accurate alignment. In the second step, we determine the sub-frame offset between two cameras over a single frame by op-timizing one parameter over the fundamental matrix, similarto [19] and [2].

Both steps assume linear motion between frames. Cameramotion is tolerated, but for the first step, it has to be smallerthan the tracked object motion.

A more detailed discussion can be found in [13].

4.1. Frame-accurate alignment

To achieve frame-accurate temporal alignment, common fea-ture points followed over multiple frames in all cameras areconsidered, e.g. a bouncing ball.

Its trajectory is initially followed by a modified versionof the KLT tracker [11, 16]. The ideal characteristic move-ment consists of extremal points of a linear motion, thereforewe use the Douglas-Peucker Algorithm [3] (with a scale pa-rameter ε, which determines the number of remaining points)for reduction, see Fig. 1(a). We adjust ε such that a sufficientnumber of intermediary points remains.

Since we are only interested in the temporal componentof the motion, not in the spatial, we can simplify our problemfurther by reducing the extremal points to binary codes, seeFig. 1(b). To this end, we model extremal points as 1 andintermediary points as 0.

Finally, we use simple string matching [14] to determinethe offset in which a motion is most similar in multiple cam-eras.

The approach requires motion that can be linearly approx-imated, and camera motion must be smaller than object mo-

Fig. 2. The cameras ca, cb, cc and cd on the sphere S togetherwith the ground plane given by g1,2,3, define the dimensionselevation and azimuth of N

tion. Under these conditions, the method described aboveis both fast and reliable, particularly when coupled with aRANSAC to remove outliers.

4.2. Subframe-accurate alignment

Given the frame accurate alignment, we choose two imagepairs T and T ′, where points in time pt and pt+1 are relatedto points p′t and p′t+1 such that pt ≤ p′t ≤ pt+1.

Furthermore, we assume that a point in time pt+a can beapproximated by a parameter a with 0 ≤ a ≤ 1 using linearinterpolation:

pt+a = (1− a)pt + a pt+1 (1)

Our basic strategy is to to find an a solving for the fun-damental matrix Ft such that the basic fundamental matrixequation

(p′t)T Ft pt = 0 (2)

is approximated by

(p′t)T Ft ((1− a)pt + a pt+1) = 0 (3)

To this end, we need 9 independent motion trajectoriesfrom each sequence, using a 9x9 matrix M(a) and an un-known 9-vector ft such that Eq. 3 becomes

M(a) ft = 0 (4)

If the constraint det(M(a)) = 0 is satisfied, i.e. a suit-able motion exists and camera movement is smaller than ob-ject movement, a unique solution for a exists and is found bysolving Eq. 4.

Given the above constraints, the offset ∆t is uniquely de-termined by a in just one single step. Accuracy can be furtherimproved by calculating the statistical mean of several suchoffsets for subsequent frames. This effectively removes out-liers.

5. IMAGE-BASED FREE VIEWPOINT VIDEO

With our Virtual Video Camera rendering system it is possi-ble to viewpoint-navigate through space and time of complexreal-world, dynamic scenes. As stated earlier our approachaccepts unsynchronized multi-video footage as input. Inex-pensive, consumer-grade camcorders suffice to acquire arbi-trary scenes, e.g., in the outdoors, without elaborate recordingsetup procedures, allowing also for hand-held recordings. In-stead of scene depth estimation, layer segmentation, or 3Dreconstruction, our approach is based on dense image cor-respondences, treating view interpolation uniformly in spaceand time: spatial viewpoint navigation, slow motion, or freeze-and-rotate effects can all be created in this unified framework[8].

The view generation system naturally extends to gener-ating stereoscopic content. The two different approaches tosynthesize a second view (see Sec. 5.2).

While we give a overview of the system in this paper, dis-cussion in great detail can found in [10].

5.1. Preprocessing

Our goal is to explore the captured scene in an intuitive wayand render a (virtual) view Iv of it, corresponding to a combi-nation of viewing direction and time. To this end, we chooseto define a 3-dimensional navigation space N that representsspatial camera coordinates as well as the temporal dimension.All new virtual viewpoints are described in respect to this nav-igation space.

To create this space cameras are placed on the surfaceof a virtual sphere, their orientations are defined by azimuthazimuth and elevation elevation (Fig. 5). Together with thetemporal dimension t, azimuth and elevation span our 3- di-mensional navigation space N . If cameras are arranged in anarc or curve around the scene, elevation is fixed, reducingNto two dimensions. A sparse set of world points provided bythe structure from motion system, is used to define a commonground plane for the navigation space.

After mapping the input images from the calibrated eu-clidean space into N , we compute a Delaunay triangulation.This enables us to partition N into a set of thetrahedra. Eachthetrahedron is defined by four vertices corresponding to fourinput images and six edges. For each edge between two im-ages Ii and Ij , we compute two dense correspondence mapsWij and Wji.

5.2. View Synthesis

Having subdivided navigation space N into tetrahedra, eachpoint v is defined by the vertices of the enclosing tetrahedronλ = {vi}, i = 1 . . . 4. Its position can be uniquely expressedas v =

∑4i=1 µivi, where µi are the barycentric coordinates

of v Fig 3. Each of the 4 vertices vi of the tetrahedron cor-responds to a recorded image Ii. Each of the 12 edges eij

Fig. 3. Determine the warping weights for the two, three andfour image case. The squares Vi denote images and the reddot the desired new viewpoint.

correspond to a correspondence map Wij, that defines a trans-lation of a pixel location x on the image plane. We are nowable to synthesize a novel image Iv for every point v insidethe recording hull of the navigation space N by multi-imageinterpolation:

Iv =

4∑i=1

µiIi, (5)

where

Ii

Πi(x) +∑

j=1,...,4,j 6=i

µjΠj(Wij(x))

= Ii(x) (6)

are the forward-warped images [12]. {Πi} defines a set ofre-projection matrices {Πi} that map each image Ii onto theimage plane of Iv , as proposed by Seitz and Dyer [15]. Thosematrices can be easily derived from camera calibration. Sincethe virtual image Iv is always oriented towards the center ofthe scene, this re-projection corrects the skew of optical axespotentially introduced by our loose camera setup and also ac-counts for jittering introduced by dynamic cameras. Imagere-projection is done on the GPU without image data resam-pling.

Up to this point the view synthesis is monocular. In or-der to create stereoscopic content a second view has to besynthesized. Since the navigation space has an azimuth pa-rameter creating the image for the second eye can be as ac-complished by synthesizing a new viewpoint from a slightlyoffset azimuth position. By varying the offset the baselinecan easily be adjusted for a pleasant S3D viewing experience.Problematic are model violations of the linear movement as-sumption underlying the 2D image correspondences. Thoseas well as misalignment of the cameras can lead to slight ver-tical disparities, which are known to cause viewer discom-fort. Another issue is, that our system is limited to view inter-polation, which makes creating new views impossible if therecording setup has no neighbouring camera on the horizontalaxis.

For the most recent production application [20] we there-fore opted for another approach [9]. The dense image cor-respondences in combination with the camera calibration canbe used to triangulate dense depth maps. Those depth maps

can in turn be used to employ depth image based rendering tocreate two stereoscopic views.

6. RECONSTRUCTING SHAPE AND MOTION

Instead of directly synthesizing the new viewpoints with theVirtual Video Camera system, the calibration and timing in-formation together with the recorded images can be used torecover geometric information. The challenge for a 3D recon-struction arises from the unconstrained nature of input data.Unsynchronized cameras in combination with dynamic scenesmake traditional static stereo, or multi-view reconstructionsinapplicable, since objects may vary their position betweenany two input images.

One of the assumptions we make is that the input videostreams show multiple views of the same scene. Since weaim to reconstruct a geometric model, we expect the scene toconsist of opaque objects with mostly diffuse reflective prop-erties. An additional assumption is made about the scene dy-namics. To be able to recover the motion, we assume a linearmotion for all points in the scene.

Our scene model represents the scene geometry as a setof small tangent plane patches. The goal is to reconstruct atangent patch for the entire visible surface. Each patch is de-scribed by its position ~c at t = 0, the normal ~n and a velocityvector ~v. The position of a patch at a given time t follows as

x(t) = ~c+ t · ~v (7)

One of the advantages of a patch based scene model is,that properties such as position, motion and visibility are in-herently defined per patch. This gives the scene model thenecessary degrees of freedom to describe complex objectswith non rigid motions. For static scenes Furukawa et al. [4]demonstrated the potential of this approaches with their Patch-based Multi-view Stereo.

6.1. The Reconstruction Process

Our algorithm starts by creating a sparse set of seed patchesin an initialization phase, and grows the seeds to cover thevisible surface by iterating expansion, optimization and filtersteps.

The algorithm processes a group of images at a time. Theimages are chosen by their respective temporal and spatialparameters. All patches extracted from an image group col-lectively form the dynamic model of the scene. This modelis valid for the time ranging from the acquisition of the firstimage of the group to the time the last selected image wasrecorded.

We start by calculating points of interest selected by eval-uating the eigenvalues from the covariation matrix of the gra-dient image. A SURF descriptor [1] is then calculated forthose points.

(a)

Fig. 4. A moving scene object captured at three timestepst0,1,2 on three image planes from the camera positions Φ0,1,2.The red dots on the image planes show the three matched fea-tures from which the path of the scene object will be recov-ered. The object path has to intersect the rays ~r0,1,2.

In a matching step features are matched across all imagesyielding multiple sets of points. The points in each set pos-sibly belonging to the same position in the scene. The set isevaluated and if possible, one patch per set is recovered.

To determine position ~c and velocity ~v, a linear equationsystem is formulated. (Fig. 4) The line of movement (7) mustintersect the viewing rays ~qi that originate from the cameracenter Φi and are cast through the image plane at the pixelposition where the patch was observed in image. The timewhen an input image Ii was recorded is denoted by ti:

Id3×3 Id3×3 · t0 −~q T

0 0 0...

.... . .

Id3×3 Id3×3 · ti 0 0 −~q Ti

·

~cT

~v T

a0

...aj

=

ΦT

0

...ΦT

i

(8)

The variables a0 to ai give the scene depth in respect tothe camera center Φi. The overdetermined linear system issolved with a SVD solver.

Using ~c, ~v and the camera calibration the reconstructedpatch is reprojected into the input images. The reprojectionerror is used to evaluate the validity of the reconstruction. Ifthe point set from which ~c and ~v were calculated has morepoints than needed for the reconstruction, RANSAC is usedto remove outliers.

If a reconstruction is found to be valid, the normal for thenew patch needs to be determined. Therefore it is initializedas pointing directly at one of the cameras Φi from which thepatch was reconstructed. To refine the normal the normalizedcross correlation (NCC) across all images is maximized usinga conjugate gradient optimization framework. In the same op-timization the position and velocity are refined. Details aboutthe cost function and the patch refinement can be found in theoriginal paper [7].

The number of patches after this initial reconstruction isvery low. To incrementally cover the entire visible surface,the existing patches are expanded along the object surfaces.

For a given patch, the surrounding pixels in the imagesare tested and if no other patch exists at this position a newpatch is created at that position. A viewing ray is cast throughthe center of the empty pixel and intersected with the planedefined by the patch position and its normal. The intersectionpoint is the center position for the newly created patch. Thevelocity and normal of the new patch are initialized with thevalues from the source patch. The new patch is then subjectedto the NCC based refinement to improve the initial values forposition ~c, velocity ~v and normal ~n.

After evaluating all patches for possible expansions, a fil-tering step is performed. The filtering eliminates inconsisten-cies in the scene model introduced by the last expansion step.Simple depth and occlusion tests are used to filter patches thatlie within or before the actual surface. A third filter test en-forces a light regularization. Therefore a similarity measureis introduced.

To determine whether two patches are similar in a givenimage, their position ~x0, ~x1 and normals ~n0, ~n1 are used toevaluate the inequality

(~x1 − ~x0) · ~n0 + (~x0 − ~x1) · ~n1 < κ. (9)

The comparison value κ is calculated from the size of the localregion in which patches are compared. If the inequality holds,the two patches are similar.

All surrounding c patches in the local region are evaluatedwith 9. The quotient of the number c′ of patches similar to theoriginal patch in relation to the total number of surroundingpatches c is the regularization criterion: c′

c < z. Our resultsshow that a quotient of z = 0.25 leads to satisfying results.

After a predefined amount of the input images is covered,the iteration process of growing and filtering the patches isstopped.

7. APPLICATIONS

To demonstrate the capabilities of the Virtual Video Cameraand the dynamic scene reconstruction we show footage froma recent music video project [20] in which our tools whereapplied in a production environment (Fig. 5).

When preparing recorded datasets for post processing themost basic decision to be made is what type of model to usefor analysing the data. The two most popular choices are a ge-ometry based scene model or image-based approaches. Tra-ditionally a lot of video and image post processing like seg-menting, compositing, color grading and even basic relight-ing is done purely in the image domain. A geometry model ishand created, or reconstructed for special effects purposes, ormore recently to create S3D content.

In the context of the Virtual Video Camera from our workflow, we use an image-based technique to generate novel view-

(a) (b) (c) (d)

(e) (f) (g) (h)

Fig. 5. (a,b) Two input views of the static set background painted with graffiti, (c) a view rendered from a position in between(a) and (b), (d) textured patches as reconstructed from the multi-view reconstruction, (e,f) input views of the foreground liveaction recording, (g) synthesized in between viewpoint masked with a color key, (h) dynamic scene reconstruction results

points. The main limitation with the current rendering ap-proach is that only interpolation between camera views ispossible. Views that leave the camera manifold can not besynthesized.

Since the underlying concept is dense image correspon-dences, that are used to warp the input material, no explicit3D model needs to be reconstructed. This allows the render-ing amorphous objects and phenomena such as fire, water orsmoke, by finding visually plausible correspondences withoutthe constraint of geometric validity.

The output material rendered from the Virtual Video Cam-era is dense by construction. When having a reconstructionapproach in mind it is noteworthy, that the results are not af-fected by low texture regions.

Although the image correspondences are not geometri-cally constrained, they can be used to estimate dense depthmaps. However we found them to be accurate enough to cre-ate high quality S3D content by depth image based rendering.By modifying the depth images before the rendering step it iseven possible to achieve a depth remapping to further enhancethe 3D experience.

Additionally only one resampling step from the recordedimages to the final view is required. This has the advantageof faithfully preserving the original scene appearance.

The major drawback of staying in the 2D image domainis, that occlusions and disocclusions are not apparent fromthe correspondence fields. It is possible to enforce symmetriccorrespondences and in case of non symmetry assume a (dis-)occlusion, in some cases this heuristic is still prone to error.Depending on the scene (no smoke, fire, water), it is possibleto use the dense depth maps as an approximation of the realscene geometry to resolve these issues. This approach yieldshigh quality results.

With the above in mind, there are cases, when a geometricreconstruction seems a better choice.

For a 3D reconstruction to give satisfying results the com-mon scene restrictions apply: only opaque objects. Glass, fogor fire, mirrors and in fact any material that has strongly viewdependent appearance will decrease the reconstruction resultas it gets harder to match across multiple images.

Besides those restrictions a 3D model as described in Sec. 6is a description of the scene holding much less redundancy.Which is an advantage when dealing with large datasets. Re-constructing data in a 3D domain instead of purely 2D im-age correspondences gives additional constraints that allowexplicit handling of occlusion and disocclusion.

When the textured 3D data is recovered, it integrates wellwith existing production tools. Tasks as relighting or com-positing other 3D content into the scene is straight forward.However to gain this data in the highest possible quality ad-ditional steps as surface reconstruction, view-dependent tex-ture mapping and error concealment during rendering becomenecessary.

As mentioned earlier depending on the scene structure ageometry reconstruction is often only possible in the salientregions of the input images. This poses a problem especiallyfor the surface reconstruction steps.

Under the given restrictions a textured scene model is stillof value, since the camera viewpoint can be chosen freely inthe scene. In contrast to the Virtual Video Camera, extrapo-lating camera positions is possible. However when movingthe synthesized viewpoint away from the original input cam-era viewpoints hole filling strategies have to be applied to fillareas where no data was observed.

8. CONCLUSION

We presented our toolchain for free-viewpoint 3D stereo video,the virtual video camera including a 3D reconstruction ap-proach. Its main strength in the context of 3D stereo video

is the flexible choice of the right tool for the desired applica-tion. Both tools the Virtual Video Camera and the dynamicscene reconstruction allow for easy post production variationof parameters such as baseline that are usually fixed with con-temporary stereo recording rigs. The low requirements for thecamera are make it cost-effective and easy to set up.

The natural limit of our system is the sparsity of camerasand frame rates. Any high speed motion of small objects orultra-wide camera spacing must be accounted for with appro-priate camera equipment.

Ultimately, we believe that post production flexibility instereo parameters including the camera position is highly de-sirable and as we have shown very well feasible.

9. ACKNOWLEDGEMENTS

This work has been funded by the European Research CouncilERC under contract No.256941 Reality CG and by the Ger-man Science Foundation, DFG MA 2555/1-3.

10. REFERENCES

[1] H. Bay, A. Ess, T. Tuytelaars, and L.V. Gool. Surf:Speeded up robust features. Computer Vision and Im-age Understanding, 110(3):346–359, 2008.

[2] Yaron Caspi, Denis Simakov, and Michal Irani. Feature-based sequence-to-sequence matching. InternationalJournal of Computer Vision, 68:53–64, 2006.

[3] D.H. Douglas and T.K. Peucker. Algorithms for the re-duction of the number of points required to representa digitized line or its caricature. Cartographica: TheInternational Journal for Geographic Information andGeovisualization, 10(2):112–122, 1973.

[4] Yasutaka Furukawa and Jean Ponce. Accurate, dense,and robust multi-view stereopsis. IEEE Trans. on Pat-tern Analysis and Machine Intelligence, 2009.

[5] M. Goesele, N. Snavely, B. Curless, H. Hoppe, and S.M.Seitz. Multi-view stereo for community photo collec-tions. In Proc. ICCV, pages 1–8, 2007.

[6] Oliver Grau, Adrian Hilton, Joe Kilner, Gregor Miller,Tim Sargeant, and Jonathan Starck. A free-viewpointvideo system for visualisation of sport scenes. SMPTEMotion Imaging, 116:213–219, 2007.

[7] Felix Klose, Christian Lipski, and Marcus Magnor. Re-constructing shape and motion from asynchronous cam-eras. In Vision, Modeling and Visualization (VMV)2010, pages 171–177, 2010.

[8] Christian Linz, Christian Lipski, Lorenz Rogge, Chris-tian Theobalt, and Marcus Magnor. Space-time visual

effects as a post-production process. In ACM Multime-dia 2010 Workshop - 1st International Workshop on 3DVideo Processing (3DVP) 2010, volume 1, pages 1–6,2010.

[9] Christian Lipski, Felix Klose, Kai Ruhl, and MarcusMagnor. Making of ”who cares?” HD stereoscopic freeviewpoint video. In Conference on Visual Media Pro-duction (CVMP) 2011, volume 8, 2011.

[10] Christian Lipski, Christian Linz, Kai Berger, Anita Sell-ent, and Marcus Magnor. Virtual video camera: Image-based viewpoint navigation through space and time.Computer Graphics Forum, 29(8):2555–2568, 2010.

[11] B.D. Lucas, T. Kanade, et al. An iterative image regis-tration technique with an application to stereo vision. InInternational joint conference on artificial intelligence,volume 3, pages 674–679. Citeseer, 1981.

[12] William Mark, Leonard McMillan, and Gary Bishop.Post-Rendering 3D Warping. In Symposium on Inter-active 3D Graphics, pages 7–16, 1997.

[13] Benjamin Meyer, Timo Stich, Marcus Magnor, andMarc Pollefeys. Subframe temporal alignment of non-stationary cameras. In Proc. British Machine VisionConference (BMVC) 2008, 2008.

[14] S.B. Needleman and C.D. Wunsch. A general methodapplicable to the search for similarities in the amino acidsequence of two proteins. Journal of molecular biology,48(3):443–453, 1970.

[15] Steven M. Seitz and Charles R. Dyer. View Morph-ing. In Proc. of ACM SIGGRAPH’96, pages 21–30, NewYork, 1996. ACM Press/ACM SIGGRAPH.

[16] Jianbo Shi and C. Tomasi. Good features to track. InComputer Vision and Pattern Recognition, 1994. Pro-ceedings CVPR ’94., 1994 IEEE Computer Society Con-ference on, pages 593 –600, 1994.

[17] A. Smolic, S. Poulakos, S. Heinzle, P. Greisen, M. Lang,A. Hornung, M. Farre, N. Stefanoski, O. Wang,L. Schnyder, R. M. Rodriguez, and M. Gross. Disparity-aware stereo 3d production tools. Conference on VisualMedia Production (CVMP), 2011.

[18] The Foundry - Ocula.http://www.thefoundry.co.uk/products/ocula/.

[19] A. Whitehead, R. Laganiere, and P. Bose. Temporal syn-chronization of video sequences in theory and in prac-tice. Applications of Computer Vision and Workshop onMotion and Video Computing, 2:132–137, 2005.

[20] Who Cares - and the Virtual Video Camera .http://graphics.tu-bs.de/projects/whocares/.