scalable 3d video of dynamic scenes

Visual Comput (2005) 21: 629–638DOI 10.1007/s00371-005-0346-7 O R I G I N A L A R T I C L E

Michael WaschbuschStephan WurmlinDaniel CottingFilip SadloMarkus Gross

Scalable 3D video of dynamic scenes

Published online: 31 August 2005© Springer-Verlag 2005

M. Waschbusch (�) · S. Wurmlin ·D. Cotting · F. Sadlo · M. GrossComputer Graphics LaboratoryDepartment of Computer ScienceSwiss Federal Institute of Technology(ETH), ZurichSwitzerland{waschbuesch, wuermlin, dcotting, sadlof,grossm}@inf.ethz.ch

Abstract In this paper we presenta scalable 3D video framework forcapturing and rendering dynamicscenes. The acquisition system isbased on multiple sparsely placed3D video bricks, each comprisinga projector, two grayscale cameras,and a color camera. Relying onstructured light with complemen-tary patterns, texture images andpattern-augmented views of thescene are acquired simultaneouslyby time-multiplexed projections andsynchronized camera exposures.Using space–time stereo on theacquired pattern images, high-qualitydepth maps are extracted, whosecorresponding surface samples are

merged into a view-independent,point-based 3D data structure. Thisrepresentation allows for effectivephoto-consistency enforcement andoutlier removal, leading to a signifi-cant decrease of visual artifacts anda high resulting rendering qualityusing EWA volume splatting. Ourframework and its view-independentrepresentation allow for simple andstraightforward editing of 3D video.In order to demonstrate its flexibility,we show compositing techniques andspatiotemporal effects.

Keywords 3D video · Free-viewpoint video · Scene acquisition ·Point-based graphics

1 Introduction

As one of the many promising emerging technologies forhome entertainment and spatiotemporal visual effects, 3Dvideo acquires the dynamics and motion of a scene dur-ing recording while providing the user with the possi-bility to change the viewpoint at will during playback.Free navigation regarding time and space in streams ofvisual data directly enhances the viewing experience andinteractivity. Unfortunately, in most existing systems vir-tual viewpoint effects have to be planned precisely andchanges are no more feasible after the scene has beenshot. As an example, Digital Air’s Movia R© systems com-prise high-speed, high-definition digital cinema camerasthat are placed accurately such that no software view inter-polation is needed. But as a consequence, postprocessingand editing possibilities are restricted.

A number of multiview video systems allow for real-istic rerenderings of 3D video from arbitrary novel view-points. However, for producing high-quality results, thecapturing systems are confined to configurations wherecameras are placed very close together. As an example,Zitnick et al. [38] covered a horizontal field of view of 30◦with 8 cameras, where only linear arrangements were pos-sible. For configurations that cover an entire hemispherewith a small number of cameras, either model-based ap-proaches need to be employed (e.g., Carranza et al. [6]with 8 cameras) or degradation in visual quality has to beaccepted (e.g., Würmlin et al. [33] with 16 cameras). Thelatter two systems are also limited by the employed recon-struction algorithms to the capture of foreground objectsor even humans only, and scalability in terms of cameraconfigurations and data structures is not addressed. More-over, the underlying representations and processes typic-ally do not allow for convenient editing.

630 M. Waschbüsch et al.

Our work is motivated by the drawbacks of the afore-mentioned systems and by the vision of bringing 3D videoto a new level where not only capturing and subsequenthigh-quality rerendering is cost-effective, convenient, andscalable, but also editing of the spatiotemporal streams iseasy to perform. We envision 3D video editing to becomeas convenient as 2D home video editing. For this pur-pose, we rely on view-independent 3D geometry streams,which allow for similar authoring and editing techniquesas carried out in common 3D content creation and mod-eling tools. Inserting novel objects to a scene or addingspatiotemporal effects is becoming straightforward withsimple postprocessing methods, and one no longer has tocope with the common limitations of image-based repre-sentations.

Specifically, we make the following contributions inthis paper:

– We introduce sparsely placed, scalable 3D video bricksthat act as low-cost z-cameras and allow simultaneoustexture and depth map acquisition using space–timestereo on structured light.

– We propose a view-independent point-based represen-tation of the depth information acquired by the 3Dvideo bricks. Different postprocessing algorithms en-able image generation from novel viewpoints with ap-pealing quality.

– We present a probabilistic rendering technique basedon view-dependent EWA volume splatting, providingclean images by smoothly blending noise from the re-construction process.

– We demonstrate the suitability of our view-independent3D representation for authoring and editing and showseveral results of 3D video, including effects like ob-ject cloning and motion trails.

2 Related work

This paper extends or integrates previous work in areaslike point-based computer graphics, depth-from-stereo,and 3D video. For the sake of conciseness, we refer thereader to the ACM SIGGRAPH 2004 course on point-based computer graphics [1] and to relevant depth-from-stereo publications [27, 37]. In the following discussion,we will confine ourselves to related work in the area of 3Dvideo.

In 3D video, multiview video streams are used torerender a time-varying scene from arbitrary viewpoints.There is a continuum of representations and algorithmssuited for different acquisition setups and applications.Purely image-based representations [18] need many denselyspaced cameras for applications like 3D-TV [21]. Dy-namic light field cameras [32, 35], which have camera

baselines of a couple of centimeters, do not need anygeometry at all. Camera configuration constraints can berelaxed by adding more and more geometry to image-based systems, as demonstrated by Lumigraphs [10].Voxel-based representations [30] can easily integrate in-formation from multiple cameras but are limited inresolution. Depth-image-based representations [2, 28] usedepth maps that are computed predominantly by stereoalgorithms [9, 35, 38]. Stereo systems still require rea-sonably small baselines and, hence, scalability and flexi-bility in terms of camera configurations is still notachieved. Redert et al. [26] use depth images acquiredby Zcams [13] for 3D video broadcast applications. Ap-propriate representations for coding 3D audio/visual dataare currently investigated by the MPEG-4 committee [29].On the other end of the continuum are model-based rep-resentations that describe the objects or the scene bytime-varying 3D geometry, possibly with additional videotextures [6, 14]. Almost arbitrary camera configurationsbecome feasible, but most existing systems are restrictedto foreground objects only.

Besides data representations, one has to distinguish be-tween online and offline applications. Matusik et al. [19,20] focus on real-time applications, e.g., 3D video con-ferencing or instant 3D replays. However, they are re-stricted to capturing foreground objects only due to thenature of their silhouette-based depth reconstruction algo-rithms. Gross et al. [11] use a 3D video system based ona point sample representation [33] for their telecollabora-tion system blue-c and share the same limitation of onlybeing able to reconstruct foreground objects. Mulligan etal. [22] also target telepresence. They compute geomet-ric models with multicamera stereo and transmit textureand depth over a network. Carranza et al. [6] present anoffline 3D video system that employs an a priori shapemodel that is adapted to the observed outline of a human.However, this system is only able to capture predefinedshapes, i.e., humans. The 3D video recorder [34] han-dles point-sampled 3D video data captured by silhouette-based reconstruction algorithms and discusses data stor-age issues. No full scene acquisition is possible with thelast two systems, but almost arbitrary camera configura-tions are possible. Zitnick et al. [38] proposed a layereddepth image representation for high-quality video viewinterpolation. Reconstruction errors at depth discontinu-ities are smoothed out by Bayesian matting. However,this approach again needs a quite dense camera setupto generate high-quality renderings in a limited viewingrange. Scalability to larger setups is not addressed by theauthors.

Cockshott et al. [7] also propose a 3D video studiobased on modular acquisition units and pattern-assistedstereo. For concurrent texture acquisition, the patterns areprojected using strobe lights requiring custom-built hard-ware. Only foreground objects are modeled using implicitsurfaces.

Scalable 3d video of dynamic scenes 631

3 Overview

Our 3D video acquisition system consists of several so-called 3D video bricks that capture high-quality depthmaps from their respective viewpoints using calibratedpairs of stereo cameras (Fig. 1). The matching algorithmused for depth extraction is assisted by projectors illu-minating the scene with binary structured light patterns.Alternating projection of a pattern and its inverse allowsfor concurrent acquisition of the scene texture using ap-propriately synchronized color cameras.

Fig. 1. Overview of 3D video framework

The depth maps are postprocessed to optimize discon-tinuities, and the results from different viewpoints are uni-fied into a view-independent, point-based scene represen-tation consisting of Gaussian ellipsoids. During merging,we remove outliers by ensuring photoconsistency of thepoint cloud with all acquired images from the texture cam-eras. Editing operations like compositing and spatiotem-poral effects can now be applied to the view-independentgeometry. Novel viewpoints of the dynamic scene are ren-dered using EWA volume splatting.

4 Scalable 3D video bricks

In this section we present the concept of our low-costz-cameras realized by 3D video bricks allowing simultan-eous acquisition of textures and depth maps.

4.1 Acquisition setup

The basic building blocks of the 3D video setup aremovable bricks containing three cameras and a projec-tor illuminating the scene with alternating patterns. Twograyscale cameras are responsible for depth extraction,while a color camera acquires the texture information ofthe scene. Figure 2 shows a single brick prototype withits components. In our current implementation, we op-erate with three bricks, each consisting of a standardPC with a genlock graphics board (NVIDIA QuadroFX3000G), a projector synchronizing to the input sig-nal (NEC LT240K), and cameras having XGA resolution

Fig. 2. 3D video brick with cameras and projector (left), simul-taneously acquiring textures (middle), and structured light patterns(right)

(Point Grey Dragonfly). The components are mounted ona portable aluminum rig as shown in Fig. 2. The systemis complemented by a synchronization microcontroller(MCU) connected to the cameras and the genlock-capablegraphics boards.

At a certain point in time, each brick can only capturedepth information from a particular fixed position. In orderto span a wider range of viewpoints and reduce occlu-sion effects, multiple movable bricks can be combined andindividually oriented to cover the desired working spaceas illustrated in Fig. 3). Scalability of multiple bricks isguaranteed because overlapping projections are explic-itly allowed by our depth reconstruction and because thecomputation load of each brick does not increase dur-ing real-time recording. Each brick performs the grabbingcompletely independently of the other bricks with the ex-ception of the frames being timestamped consistently byusing a common synchronization device.

Fig. 3. Configuration of our 3D video prototype system

In order to compute valid depth maps and merge theinformation gained from several bricks, all cameras inthe 3D video system must be calibrated intrinsically andextrinsically. We determine imaging properties of all cam-eras using the MATLAB camera calibration toolbox [3].The projectors do not need to be calibrated.

4.2 Simultaneous texture and depth acquisition

Each brick concurrently acquires texture information withthe color camera and depth information using the stereo


Fig. 4. Camera exposure with inverse pattern projection

pair of grayscale cameras. Stereovision (Sect. 4.3) gener-ally requires a highly texturized scene to find good corre-lations between different views. It generally fails in recon-structing simple geometry of uniformly colored objects,e.g., white walls. Additionally, the textures should be non-periodic to guarantee unique matches. As a consequence,we add artificial textures to the scene by projecting struc-tured light patterns, as originally proposed by Kang etal. [16]. We use a binary vertical stripe pattern with ran-domly varying stripe widths. It supports strong and uniquecorrelations in the horizontal direction and is at the sametime insensitive to vertical deviations that may occur frominaccuracies in the camera calibration. To avoid untextur-ized shadows, the scene is illuminated by patterns from allbricks at the same time. Our approach has the advantageof being insensitive to interferences between the differentprojections and does not need a projector calibration, un-like pure structured light approaches or stereo matchingbetween cameras and projectors.

Alternating projections of structured light patterns, andthe corresponding inverses allow for simultaneous acqui-sition of the scene textures using an appropriately syn-chronized texture camera as illustrated in Fig. 4. Note thatthis camera does not see the patterns emanating from theprojector, but only a constant white light, which preservesthe original scene texture (Fig. 2).

Since the patterns are changing at a limited rate of60 Hz (projector input frequency), flickering is slightlyvisible to the human eye. Alternative solutions using im-perceptible structured light [8] do not show any flickering,but require faster, more sensitive, and, therefore, more ex-pensive cameras for reliable stereo depth extraction.

4.3 Stereo matching on structured light

Each brick acquires the scene geometry using a depth-from-stereo algorithm. Depth maps are computed for theimages of the left and right grayscale cameras by search-ing for corresponding pixels. To reduce occlusion prob-lems between the views, the cameras are mounted ata small horizontal baseline of 20 cm.

In recent decades, a large variety of stereo algorithmshas been developed. A survey of different methods andtheir implementations can be found in Scharstein etal. [27]. Recently, Zitnick et al. [38] used a segmentation-based algorithm to generate high-quality 3D video. Ex-periments have shown that the approach works well for

conventional passive stereo but fails on structured lightpatterns, where segments seem to become too small andtoo similar to result in unique correlations. According tothe authors, their method is more suited for multibaselinestereo applications.

Our work is based on space–time stereo [36, 37] thatexploits time coherence to correlate the stereo images andcomputes disparities with subpixel accuracy. The authorsformulate stereo matching as a maximization problemover an energy E(d, du, dv, dt) that defines a matchingcriterion of two pixel correlation windows. The solutiondelivers both the disparity d and its derivatives du, dv inimage space and dt in the time domain. In contrast tothe original work, we employ the maximum normalizedcross correlation (MNCC) as similarity measure, whichis robust against global color variations between the leftand right image. In our implementation, we optimize Eusing the downhill-simplex method. To prevent the algo-rithm from converging to a local minimum, we use thedisparities of neighboring pixels as a starting guess for theoptimization and additionally repeat the process with sev-eral random starting values. We consider the maximumvalue of E as a measure of the correlation quality andreject all disparities with an energy below a certain thresh-old.

The number of potential outliers can be decreased byextending the correlation window in the temporal dimen-sion to cover three or more images. However, becausecorrelation assumes continuous surfaces, there arise someartifacts at depth discontinuities. For moving scenes, dis-continuities in the image space are extended into the tem-poral domain, making correlation computation even moredifficult. For complex dynamic scenes, we therefore usean adaptive correlation window covering multiple timesteps only in static parts of the images that can be de-tected by comparing successive frames. Remaining errorsare smoothed out with our discontinuity optimization ap-proach presented in the next section.

4.4 Discontinuity optimization

We smooth discontinuity artifacts by applying a two-phasepostprocessing to the disparity images. First, we conser-vatively identify the regions of wrong disparities. Second,we extrapolate new disparities into these regions fromtheir neighborhoods.

In the first phase, the detection of occluded parts is per-formed by a simple cross checking. We perform the stereocorrelation twice, between the left and right image andvice versa. Corresponding pixels in both disparity mapsshould have the same values, otherwise they belong toocclusions. This way we mask out all pixels whose dispar-ities differ about more than one pixel.

The second phase operates on depth images computedfrom the disparity maps. We detect remaining outliers and


fill all missing depths by extrapolating from their neigh-bors. To recover the correct discontinuities we block theextrapolation at texture edges of the color images. As thecolor textures are acquired by another camera, we firstwarp the depth map into the texture camera. Then, we de-compose the texture image into segments of similar colorsusing simple color quantization. With a high probability,all pixels in one color segment belong to one continuoussurface and therefore have similar depths. Differing depthscan be considered as outliers, which are identified by clus-tering the depths in each color segment: the biggest clusteris assumed to represent the correct surface; pixels of allsmaller clusters are removed. The holes in the depth mapare then filled by a moving least squares extrapolation [17]constrained by the current color segment.

Figure 5 shows a comparison of the original depthimage with an image computed by our discontinuity op-timization algorithm. For illustration we embedded theedges of the texture images. Notice how holes are filledand the depth discontinuities tightly fit to the color edges.Nevertheless, errors in the color segmentation can stilllead to some outliers at discontinuities. However, most ofthem are eliminated during view merging as discussed inSect. 5.2.

Fig. 5. Discontinuity optimization. Upper left: color image, up-per right: color segmentation. Comparison of original depth image(lower left) with the one after our discontinuity optimization (lowerright). For illustration, color edges have been embedded

5 View-independent scene representation

To model the resulting 3D scene, we propose a view-independent, point-based data representation. By merg-ing all reconstructed views into a common world refer-ence frame, we achieve a convenient, scalable representa-tion: additional views can be added very easily by back-

projecting their image pixels. Our model is in principlecapable of providing a full 360◦ view if the scene hasbeen acquired from enough viewpoints. Unlike image-based structures, it is possible to keep the amount ofdata low by removing redundant points from the geom-etry [24]. Compared to mesh-based methods, points pro-vide advantages in terms of scene complexity becausethey reduce the representation to the absolutely neces-sary data and do not carry any topological information,which is often difficult to acquire and maintain. As eachpoint in our model has its own assigned color, we alsodo not have to deal with texturing issues. Moreover, ourview-independent representation is very suitable for 3Dvideo-editing applications since tasks like object selectionor relighting can be achieved easily with standard point-processing methods [1].

5.1 Point-based data model

Our point-based model consists of an irregular set of sam-ples, where each sample corresponds to a point on a sur-face and describes its properties such as location and color.The samples can be considered as a generalization of con-ventional 2D image pixels toward 3D video. If required,the samples can be easily extended with additional at-tributes like surface normals for relighting.

To avoid artifacts in rerendering, we have to ensure fullsurface coverage of the samples. Thus, our samples cannotbe represented by infinitesimal points but need to be con-sidered as small surface or volume elements. One obviousrepresentation are surfels [25], which are small ellipticaldisks aligned tangentially to the surface. However, surfelsdo not handle noise due to inaccurate 3D reconstruction orcamera calibration very well and require accurate geome-tries and therefore stable surface normals.

Therefore, we have chosen a different approach, simi-lar to that of Hofsetz et al. [12]. Every point is modeled bya 3D Gaussian ellipsoid spanned by the vectors t1, t2, andt3 around its center p. This corresponds to a probabilisticmodel describing the positional uncertainty of each pointby a trivariate normal distribution

pX(x) = N(x; p, V)

= 1√

(2π)3|V|e− 12 (x−p)TV−1(x−p) , (1)

with expectation value p and covariance matrix

V = ΣT ·Σ = (t1 t2 t3

)T · (t1 t2 t3 ,)

(2)

composed of 3×1 column vectors ti .To estimate V, Hofsetz et al. [12] have chosen an ap-

proach based on the quality of the pixel correlation of thestereo matching. It turns out that these resulting heuristicuncertainties are quite large compared to the high-quality


Fig. 6. Construction of a 3D Gaussian ellipsoid

disparities we are able to obtain from our structured-light-assisted approach. Consequently, we propose a differentapproach that constrains the uncertainties to cover onlysmall but well-defined acquisition errors. We assume thatmost disparities are correctly estimated up to small errorscaused by deviations in the camera calibration and com-pute point sizes that just provide full surface coverage.

Assuming a Gaussian model for each image pixel un-certainty, we first compute the back-projection of the pixelinto three-space, which is a 2D Gaussian parallel to theimage plane spanned by two vectors tu and tv. Extrusioninto the third domain by adding a vector tz guaranteesa full surface coverage under all possible views. This isillustrated in Fig. 6.

Each pixel (u, v) is spanned by orthogonal vectorsσu(1, 0)T and σv(0, 1)T in the image plane. Assuminga positional deviation σc, the pixel width and height underuncertainty are σu = σv = 1+σc. σc is estimated to be theaverage reprojection error of our calibration routine.

The depth z of each pixel is inversely proportional toits disparity d as defined by the equation

z = − fL · ‖cL − cR‖d + pL − pR

, (3)

where fL is the focal length of the left rectified camera, cLand cR are the centers of projection, and pL and pR the u-coordinates of the principal points. The depth uncertaintyσz is obtained by differentiating Eq. 3 and augmenting thegradient ∆d of the disparity with its uncertainty σc:

σz = fL · ‖cL − cR‖(d + pL − pR)2 · (∆d +σc) . (4)

Now, we can construct for each pixel its Gaussian inray space with

ΣR =(

σu · z 0 σz ·u0 σv · z σz ·v0 0 σz

)

. (5)

This is transformed into the world coordinate system by

Σ = P−1 ·ΣR (6)

using the camera projection matrix P.The centers p of the ellipsoids are constructed by back-

projection as

p = P−1 · (u, v, 1)T · z + c , (7)

where c is the center of projection of the camera.

5.2 Photoconsistency enforcement

After back-projection, the point model still contains out-liers and falsely projected samples. Some points originat-ing from a specific view may look wrong from extrap-olated views due to reconstruction errors, especially atdepth discontinuities. In the 3D model, they may covercorrect points reconstructed from other views, disturbingthe overall appearance of the 3D video. Thus, we removethose points by checking the whole model for photocon-sistency with all texture cameras.

After selecting a specific texture camera, we succes-sively project each ellipsoid (as computed in Sect. 5.1)into the camera image in increasing depth order, startingwith the points closest to the camera. We determine allpixels of the original image that are covered by the pro-jection and not yet occluded by previously tested, validellipsoids. We compare the average color of those pixelswith the color of the ellipsoid. If both colors differ toomuch, the point sample is removed. Otherwise, we raster-ize the ellipsoid into a z-buffer that is used for occlusiontests for all subsequent points.

As a result, enforcing photoconsistency considerablyimproves the seamless fit of multiple acquired depth mapsin our model. The reduction of artifacts can be clearlyseen in Fig. 7. Nevertheless, there remain some issues withmixed pixels, i.e., silhouette pixels possessing a color in-terpolated from different surfaces. These tend to produceholes in the cleaned model. This may be solved usingboundary-matting techniques [15]. Currently, we applyour consistency check conservatively and tolerate remain-ing outliers which are not detected.

Fig. 7. Enforcing photo consistency during view merging: Without(left) and with (right) enforcement


6 Rendering

We render novel viewpoints of the scene using the GPUand CPU cooperatively. Smooth images are generatedusing the uncertainties of the Gaussian ellipsoids. Ourmethod combines the advantages of two probabilisticimage generation approaches described in Broadhurst etal. [4]. Additionally we perform a view-dependent blend-ing similar to Hofsetz et al. [12].

6.1 Probabilistic rendering

Broadhurst et al. [4] use probabilistic volume ray castingto generate smooth images. Each ray is intersected withthe Gaussians of the scene model. At a specific intersec-tion point x with sample i , the evaluation N(x; pi; Vi)of the Gaussian describes the probability that a ray willhit the corresponding surface point. To compute the finalpixel color, two different approaches are described. Themaximum likelihood method associates a color with theray using only the sample with the most probable inter-section. The second approach employs the Bayes rule: Itintegrates all colors along each ray weighted by the proba-bilities without considering occlusions. Thus, the color ofa ray R is computed as

cR =∫

x∈R

∑i ci N(x; pi, Vi)∫

x∈R

∑i N(x; pi, Vi)

. (8)

The maximum likelihood method generates crisp im-ages, but it also sharply renders noise in the geometry. TheBayesian approach produces very smooth images withfewer noise but is incapable of handling occlusions andrendering solid surfaces in an opaque way.

We propose a rendering method that combines bothapproaches in order to benefit from their respective advan-tages. Our idea is to accumulate the colors along each rayas in the Bayesian setting but to stop as soon as a max-imum accumulated probability has been reached. Reason-ably, a Gaussian sample should be completely opaque ifthe ray passes its center. The line integral through the cen-ter of a 3D Gaussian has a value of 1

2πand for any ray R it

holds that∫

x∈R

N(x; p, V) ≤ 1

2π. (9)

Thus, we accumulate the solution of the integrals of Eq. 8by traversing along the ray from the camera into the sceneand stop as soon as the denominator of Eq. 8 reaches 1

2π.

Assuming that solid surfaces are densely sampled, theprobabilities within the surface boundaries will be highenough so that the rays will stop within the front sur-face.

Fig. 8. Comparison of maximum likelihood (left) and Bayesian ren-dering (center) with our approach (right)

We compare the maximum likelihood and Bayesianrendering with our approach on noisy data in Fig. 8. No-tice the large distortions in the maximum likelihood imagethat get smoothed out by the other two methods. How-ever, the Bayesian renderer blends all the points includ-ing those from occluded surfaces, while our method ren-ders opaque surfaces and maintains the blending. Thus,our renderer provides the advantages of both previousmethods.

In our implementation, we replace the ray caster bya volume splatter [39] running on graphics hardware.After presorting the Gaussians according to their depthsby the CPU, the GPU splats them from front to back. Thepixel colors are blended according to the Gaussian alphamasks until the accumulated alphas reach a level of satu-ration. This is directly supported by the OpenGL blendingfunction GL_SRC_ALPHA_SATURATE.

6.2 View-dependent blending

One specific sample usually looks most accurate from theview it has been acquired from. As the angle between theacquisition and the virtual view becomes larger, the qual-ity decreases depending on the depth uncertainty of theGaussian. Projections of samples with high uncertaintybecome more and more stretched, introducing visible ar-tifacts, while samples with low uncertainties look goodfrom all views. We treat this issue by applying the view-dependent blending of Hofsetz et al. [12]. The authorscompute an alpha value representing the maximum opac-ity of each Gaussian in its center using the view-dependentcriteria of Buehler et al. [5] weighted by the individualdepth uncertainty σz.

Compared to a conventional surfel-based approach ourcombination of blending methods is able to better smoothout geometry noise in the model. This is clearly visible inFig. 9.

7 Results and discussion

For the results presented in this section, we have recordeda dynamic scene with our setup consisting of threesparsely placed bricks covering an overall viewing angleof 70◦ horizontally and 30◦ vertically. Figure 10 shows


Fig. 9. Rendering using surfels (left) and our view-dependent uncer-tainty blending (right)

novel views of the acquired scene in Fig. 2, rendered fromour reconstructed 3D model.

Our rerenderings have a decent look with a high-quality texture. Acquisition noise is smoothed out byour blending method. We are even able to reconstructhighly detailed geometry like the folds in the tableclothshown in Fig. 11. However, there are still some artifactsat silhouettes that we would like to eliminate in the fu-

Fig. 10. Re-renderings of the 3D video from novel viewpoints

Fig. 11. Geometric detail in the tablecloth. For illustration we re-computed smooth surface normals and rendered the scene withPhong lighting under two different illumination conditions

Fig. 12. Special effects: actor cloning (left), motion trails (right)

ture. This is possible by using matting approaches asdone by Zitnick et al. [38]. Some remaining outliers arealso visible in the images. They could be reduced usinga combination of multiple outlier removal algorithms [31]and by enforcing time coherence in the whole recon-struction pipeline. Big clusters of outliers tend to growin our discontinuity optimization stage if they dominatethe correct depths in a color segment. We are investi-gating hierarchical segmentation approaches as a pos-sible solution. Furthermore, enforcing time coherencemay help in filling remaining holes caused by occlu-sions.

With our system we are able to acquire a large viewingrange with a relatively low amount of cameras. To sup-port increasingly large ranges, our system is scalable up tofull spherical views. To fully cover 360◦ in all dimensionsabout 8 to 10 3D video bricks are needed. Note that thisis not constrained to convex views. Although overlaps inthe geometry can help to improve the overall quality dueto the photoconsistency enforcement, they are not requiredas each brick reconstructs its own scene part indepen-dently.

The use of projectors still imposes some practical con-straints because of the visible light spots and shadowsthat are created in the scene. We accept this limitation forthe sake of a maximum 3D reconstruction quality. Usingcalibrated projectors it would be possible to compute theincident light at each surface point and compensate for theartifacts.

Our view-independent data model provides possibil-ities for novel effects and 3D video editing. Due to itspoint-based structure we are able to employ any kind ofavailable point processing algorithms [1]. Once the 3D in-formation is available, selection and compositing issuesbecome straightforward and can be easily implementedusing spatial clustering or bounding box algorithms. Suchtasks are much harder to achieve on both conventional 2Dvideo and view-dependent 3D video approaches based onlight fields or depth maps only. Apart from the well-knowntime freeze we show two example effects in Fig. 12. Weclone the actor by copying its corresponding point cloudto other places in the scene. Motion trails are generatedby compositing semitransparent renderings of moving ob-jects from previous timesteps.


8 Conclusions and future work

We presented a system for recording, processing, andrerendering 3D video of dynamic scenes. We are able toobtain high-quality depth maps using space–time stereoon structured light while concurrently acquiring texturesof the scene. The brick concept combined with a view-independent data model allows for scalable capturing ofa large viewing range with sparsely placed components.Decent-quality images of novel views are achieved usingGaussian ellipsoid rendering with view-dependent blend-ing methods. Our point-based, view-independent data rep-resentation is well suited for spatiotemporal video editing.The representation can directly benefit from a large var-iety of available point-based processing algorithms, e.g.,normal estimation [23] for relighting effects or simplifi-cation [24] for further redundancy elimination or level-of-detail rendering.

In the future, we will further investigate the editingcapabilities of our representation. The ultimate goal is toprovide a tool for spatiotemporal editing that is as easy touse as today’s 2D video software and at the same time pro-vides all novel possibilities that arise from a time-varying3D scene representation. Furthermore, we would like toimprove the resulting image quality by exploiting inter-brick correlation and eliminating remaining artifacts atsilhouettes using matting approaches. It would also be de-sirable to achieve a comparable rendering quality usingpassive reconstruction algorithms only, which would makeour system suitable for large outdoor scenes. In addition,we want to address compression of time-varying point-sampled 3D video streams.

Acknowledgement We would like to thank Stefan Rondinelli forimplementing the stereo algorithm and Tim Weyrich for the fruitfuldiscussions. This work is carried out in the context of the blue-c-II project, funded by ETH Grant No. 0-21020-04 as an internalpolyproject.

References1. Alexa, M., Gross, M., Pauly, M., Pfister,

H., Stamminger, M., Zwicker, M.:Point-Based Computer Graphics.SIGGRAPH ’04 Course Notes (2004)

2. Bayakovski, Y., Levkovich-Maslyuk, L.,Ignatenko, A., Konushin, A., Timasov, D.,Zhirkov, A., Han, M., Park, I.K.: Depthimage-based representations for static andanimated 3D objects. In: ICIP ’02, 3, 25–28(2002)

3. Bouguet, J.Y.: Camera calibration toolboxfor matlab,http://www.vision.caltech.edu/bouguetj/calib_doc

4. Broadhurst, A., Drummond, T., Cipolla, R.:A probabilistic framework for the spacecarving algorithm. In: ICCV ’01,pp. 388–393 (2001)

5. Buehler, C., Bosse, M., McMillan, L.,Gortler, S., Cohen, M.: Unstructuredlumigraph rendering. In: SIGGRAPH ’01,pp. 425–432 (2001)

6. Carranza, J., Theobalt, C., Magnor, M.,Seidel, H.P.: Free-viewpoint video ofhuman actors. In: SIGGRAPH ’03,pp. 569–577 (2003)

7. Cockshott, W.P., Hoff, S., Nebel, J.C.: Anexperimental 3D digital TV studio. In:Vision, Image & Signal Processing ’03,pp. 28–33 (2003)

8. Cotting, D., Naef, M., Gross, M., Fuchs,H.: Embedding imperceptible patterns intoprojected images for simultaneousacquisition and display. In: ISMAR ’04,pp. 100–109 (2004)

9. Goldlücke, B., Magnor, M., Wilburn, B.:Hardware-accelerated dynamic light fieldrendering. In: VMV ’02, pp. 455–462(2002)

10. Gortler, S.J., Grzeszczuk, R., Szeliski, R.,Cohen, M.F.: The lumigraph. In:SIGGRAPH ’96, pp. 43–54 (1996)

11. Gross, M., Würmlin, S., Näf, M.,Lamboray, E., Spagno, C., Kunz, A.,Moere, A.V., Strehlke, K., Lang, S.,Svoboda, T., Koller-Meier, E., Gool, L.V.,Staadt, O.: blue-c: A spatially immersivedisplay and 3D video portal fortelepresence. In: SIGGRAPH ’03,pp. 819–827 (2003)

12. Hofsetz, C., Ng, K., Max, N., Chen, G.,Liu, Y., McGuinness, P.: Image-basedrendering of range data with estimateddepth uncertainty. IEEE CG&A 24(4),34–42 (2005)

13. Iddan, G.J., Yahav, G.: 3D imaging in thestudio (and elsewhere . . . ). In: SPIE ’01,4298, 48–55 (2001)

14. Kanade, T., Rander, P., Narayanan, P.J.:Virtualized reality: construction of virtualworlds from real scenes. IEEE Multimedia4(1), 34–47 (1997)

15. Kang, S., Szeliski, R.: Boundary mattingfor view synthesis. In: CVPRW ’04(2004)

16. Kang, S., Webb, J., Zitnick, L., Kanade, T.:A multi-baseline stereo system with activeillumination and real-time imageacquisition. In: ICCV ’95, pp. 88–93 (1995)

17. Levin, D.: Mesh-independent surfaceinterpolation. In: Geometric Modeling forScientific Visualization, pp. 37–49, ed. byBrunnett, Hamann, Mueller, Springer 2003

18. Levoy, M., Hanrahan, P.: Light fieldrendering. In: SIGGRAPH ’96, pp. 31–42(1996)

19. Matusik, W., Buehler, C., McMillan, L.:Polyhedral visual hulls for real-timerendering. In: EGRW ’01, pp. 115–125(2001)

20. Matusik, W., Buehler, C., Raskar, R.,Gortler, S.J., McMillan, L.: Image-basedvisual hulls. In: SIGGRAPH ’00,pp. 369–374 (2000)

21. Matusik, W., Pfister, H.: 3D TV: A scalablesystem for real-time acquisition,transmission, and autostereoscopic displayof dynamic scenes. In: SIGGRAPH ’04(2004)

22. Mulligan, J., Daniilidis, K.:View-independent scene acquisition fortele-presence. In: International Symposiumon Augmented Reality, pp. 105–110 (2000)

23. Pauly, M., Gross, M.: Spectral processingof point sampled geometry. In:SIGGRAPH ’01(2001)

24. Pauly, M., Gross, M., Kobbelt, L.: Efficientsimplification of point-sampled geometry.In: VIS ’02, pp. 163–170 (2002)

25. Pfister, H., Zwicker, M., van Baar, J.,Gross, M.: Surfels: Surface elements asrendering primitives. In: SIGGRAPH ’00,pp. 335–342 (2000)

26. Redert, A., de Beeck, M.O., Fehn, C.,Ijsselsteijn, W., Pollefeys, M., Gool, L.V.,Ofek, E., Sexton, I., Surman, P.: ATTEST:Advanced three-dimensional televisionsystem technologies. In: 3DPVT ’02,pp. 313–319 (2002)

27. Scharstein, D., Szeliski, R.: A taxonomyand evaluation of dense two-frame stereocorrespondence algorithms. Int. J. Comput.Vis. 47(1–3), 7–42 (2002)

28. Shade, J., Gortler, S., He, L.W., Szeliski,R.: Layered depth images. In:SIGGRAPH ’98, pp. 231–242 (1998)

29. Smolic, A., Kimata, H.: Description ofexploration experiments in 3DAV. In:JTC1/SC29/WG11 N6194. ISO/IEC(2003)

30. Vedula, S., Baker, S., Kanade, T.:Spatio-temporal view interpolation. In:EGRW ’02, pp. 65–76 (2002)

31. Weyrich, T., Pauly, M., Keiser, R., Heinzle,S., Scandella, S., Gross, M.:Post-processing of scanned 3D surface


data. In: Eurographics Symposium onPoint-Based Graphics ’04 (2004)

32. Wilburn, B., Joshi, N., Vaish, V., Talvala,E.V., Antunez, E., Barth, A., Adams, A.,Horowitz, M., Levoy, M.: Highperformance imaging using large cameraarrays. In: SIGGRAPH ’05 pp. 765–776(2005)

33. Würmlin, S., Lamboray, E., Gross, M.: 3Dvideo fragments: Dynamic point samplesfor real-time free-viewpoint video. In:Computers and Graphics ’04 28(1), 3–14(2004)

34. Würmlin, S., Lamboray, E., Staadt, O.G.,Gross, M.H.: 3D video recorder. In:Proceedings of Pacific Graphics 2002,pp. 325–334. IEEE Press, New York (2002)

35. Yang, J.C., Everett, M., Buehler, C.,McMillan, L.: A real-time distributed lightfield camera. In: EGRW ’02, pp. 77–86(2002)

36. Zhang, L., Curless, B., Seitz, S.M.:Spacetime stereo: shape recovery fordynamic scenes. In: CVPR ’03,pp. 367–374 (2003)

37. Zhang, L., Snavely, N., Curless, B., Seitz,S.M.: Spacetime faces: high resolutioncapture for modeling and animation. In:SIGGRAPH ’04, pp. 548–558 (2004)

38. Zitnick, C.L., Kang, S.B., Uyttendaele, M.,Winder, S., Szeliski, R.: High-quality videoview interpolation using a layeredrepresentation. In: SIGGRAPH ’04,pp. 600–608 (2004)

39. Zwicker, M., Pfister, H., van Baar, J.,Gross, M.: EWA splatting. IEEE Trans.Visual. Comput. Graph. 8(3), 223–238(2002)

MICHAEL WASCHBÜSCH is currently a Ph.D.candidate in the Computer Graphics Laboratoryat ETH Zurich. In 2003, he received his com-puter science diploma degree from the Univer-sity of Kaiserslautern, Germany. His researchinterests include 3D video, 3D reconstruction,point-based rendering, and graphics hardware.

STEPHAN WÜRMLIN is currently a postdoctoralresearcher in the Computer Graphics Labora-tory and project leader of the blue-c-II project(http://blue-c-II.ethz.ch). He re-ceived his Ph.D. from ETH Zurich in 2004 onthe design of the 3D video technology for theblue-c collaborative virtual reality system. Hiscurrent research interests include free-viewpointvideo, point-based representations and render-ing, real-time rendering, virtual reality, andmultimedia coding.

DANIEL COTTING received his computer sci-ence diploma degree from ETH Zurich. He iscurrently enrolled in a Ph.D. program at the ETHComputer Graphics Laboratory led by Prof. Dr.Markus Gross. Daniel’s current research inter-ests include projection and display technologies,imperceptible structured light, augmented reality,interaction, and 3D reconstruction.

FILIP SADLO is a Ph.D. candidate in computerscience at the Computer Graphics Laboratory ofETH Zurich, where he received his diploma in2003. His research interests include scientific vi-sualization, 3D reconstruction, and imaging.

MARKUS GROSS is a professor of computer sci-ence and director of the Computer Graphics Lab-oratory of ETH Zurich. He received a Master ofScience in electrical and computer engineeringand a Ph.D. in computer graphics and image an-alysis, both from the University of Saarbrucken,

Germany. From 1990 to 1994 Dr. Gross workedfor the Computer Graphics Center in Darmstadt,where he established and directed the VisualComputing Group. His research interests includepoint-based graphics, physics-based modeling,multiresolution analysis, and virtual reality. Hehas been widely publishing and lecturing oncomputer graphics and scientific visualization,and he authored the book Visual Computing,Springer, 1994. Dr. Gross has taught courses atmajor graphics conferences including ACM SIG-GRAPH, IEEE Visualization, and Eurographics.He is the associate editor of IEEE ComputerGraphics and Applications and has served asa member of international program committeesof many graphics conferences. Dr. Gross hasbeen a papers cochair of the IEEE Visualization’99, Eurographics 2000, and IEEE Visualization2002 conferences. He is currently chair of thepapers committee of ACM SIGGRAPH 2005.

scalable 3d video of dynamic scenes

Documents