Multiview ReconstructionMultiview Reconstruction

Why More Than 2 Views?Why More Than 2 Views?

• BaselineBaseline– Too short – low accuracyToo short – low accuracy

– Too long – matching becomes hardToo long – matching becomes hard

Why More Than 2 Views?Why More Than 2 Views?

• Ambiguity with 2 viewsAmbiguity with 2 views

Camera 1Camera 1 Camera 2Camera 2Camera 3Camera 3

Trinocular StereoTrinocular Stereo

• Straightforward approach to eliminate bad Straightforward approach to eliminate bad correspondencescorrespondences– Pick 2 views, find correspondencesPick 2 views, find correspondences

– For each matching pair, reconstruct 3D pointFor each matching pair, reconstruct 3D point

– Project point into 3Project point into 3rdrd image image

– If can’t find correspondence near predicted If can’t find correspondence near predicted location, rejectlocation, reject

Volumetric Multiview ApproachesVolumetric Multiview Approaches

• Goal: find a model consistent with imagesGoal: find a model consistent with images

• ““Model-centric” (vs. image-centric)Model-centric” (vs. image-centric)

• Typically use discretized volume (voxel Typically use discretized volume (voxel grid)grid)

• For each voxel, compute occupied / freeFor each voxel, compute occupied / free(for some algorithms, also color, etc.)(for some algorithms, also color, etc.)

Photo ConsistencyPhoto Consistency

• Result: not necessarily correct sceneResult: not necessarily correct scene

• Many scenes produce the same imagesMany scenes produce the same images

All scenesAll scenes

Photo-consistent scenesPhoto-consistent scenes

True sceneTrue scene ReconstructedReconstructedscenescene

Silhouette CarvingSilhouette Carving

• Find silhouettes in all imagesFind silhouettes in all images

• Exact version:Exact version:– Back-project all silhouettes, find intersectionBack-project all silhouettes, find intersection

Binary ImagesBinary Images

Silhouette CarvingSilhouette Carving

• Find silhouettes in all imagesFind silhouettes in all images

• Exact version:Exact version:– Back-project all silhouettes, find intersectionBack-project all silhouettes, find intersection

Silhouette CarvingSilhouette Carving

• Discrete version:Discrete version:– Loop over all voxels in some volumeLoop over all voxels in some volume

– If projection into images lies inside all If projection into images lies inside all silhouettes, mark as occupiedsilhouettes, mark as occupied

– Else mark as freeElse mark as free

Silhouette CarvingSilhouette Carving

Voxel ColoringVoxel Coloring

• Seitz and Dyer, 1997Seitz and Dyer, 1997

• In addition to free / occupied, store colorIn addition to free / occupied, store colorat each voxelat each voxel

• Explicitly accounts for occlusionExplicitly accounts for occlusion

Voxel ColoringVoxel Coloring

• Basic idea: sweep through a voxel gridBasic idea: sweep through a voxel grid– Project each voxel into each image Project each voxel into each image in whichin which

it is visibleit is visible

– If colors in images agree, mark voxel with colorIf colors in images agree, mark voxel with color

– Else, mark voxel as emptyElse, mark voxel as empty

• Agreement of colors based on comparing Agreement of colors based on comparing standard deviation of colors to thresholdstandard deviation of colors to threshold

Voxel Coloring and OcclusionVoxel Coloring and Occlusion

• Problem: which voxels are visible?Problem: which voxels are visible?

• Solution, part 1: constrain camera viewsSolution, part 1: constrain camera views– When a voxel is considered, necessary occlusion When a voxel is considered, necessary occlusion

information must be availableinformation must be available

– Sweep occluders before occludeesSweep occluders before occludees

– Constrain camera positions to allow this sweepConstrain camera positions to allow this sweep

Voxel Coloring Sweep OrderVoxel Coloring Sweep Order

LayersLayers

SceneScene

TraversalTraversal

SeitzSeitz

Voxel Coloring Camera PositionsVoxel Coloring Camera Positions

Inward-lookingCameras above scene

Outward-lookingCameras inside scene

SeitzSeitz

Panoramic Depth OrderingPanoramic Depth Ordering

• Cameras oriented in many different directionsCameras oriented in many different directions

• Planar depth ordering does not applyPlanar depth ordering does not apply

SeitzSeitz

Panoramic Depth OrderingPanoramic Depth Ordering

Layers radiate outwards from camerasLayers radiate outwards from camerasSeitzSeitz

Panoramic Depth OrderingPanoramic Depth Ordering

SeitzSeitzLayers radiate outwards from camerasLayers radiate outwards from cameras

Panoramic Depth OrderingPanoramic Depth Ordering

SeitzSeitzLayers radiate outwards from camerasLayers radiate outwards from cameras

Voxel Coloring and OcclusionVoxel Coloring and Occlusion

• Solution, part 2: per-image mask of which Solution, part 2: per-image mask of which pixels have been usedpixels have been used– Each pixel only used onceEach pixel only used once

– Mask filled in as sweep progressesMask filled in as sweep progresses

Image AcquisitionImage Acquisition

•Calibrated TurntableCalibrated Turntable

•360° rotation (21 images)360° rotation (21 images)

Selected Dinosaur ImagesSelected Dinosaur Images

Selected Flower ImagesSelected Flower Images SeitzSeitz

Voxel Coloring ResultsVoxel Coloring Results

Dinosaur ReconstructionDinosaur Reconstruction72 K voxels colored72 K voxels colored7.6 M voxels tested7.6 M voxels tested7 min. to compute 7 min. to compute on a 250MHz SGIon a 250MHz SGI

Flower ReconstructionFlower Reconstruction70 K voxels colored70 K voxels colored7.6 M voxels tested7.6 M voxels tested7 min. to compute 7 min. to compute on a 250MHz SGIon a 250MHz SGI

SeitzSeitz

Voxel Coloring ResultsVoxel Coloring Results

• With texture: good resultsWith texture: good results

• Without texture: regions tend to “bulge Without texture: regions tend to “bulge out”out”– Voxels colored at earliest time at which Voxels colored at earliest time at which

projection into images is consistentprojection into images is consistent

– Model good for re-rendering: image will look Model good for re-rendering: image will look correct for viewpoints near the original onescorrect for viewpoints near the original ones

Limitations of Voxel ColoringLimitations of Voxel Coloring

• A view-independent depth orderA view-independent depth ordermay not existmay not exist

• Need more powerful general-case Need more powerful general-case algorithmsalgorithms– Unconstrained camera positionsUnconstrained camera positions

– Unconstrained scene geometry/topologyUnconstrained scene geometry/topology

pp qq

Space CarvingSpace Carving

Image 1Image 1 Image NImage N

……......

Initialize to a volume V containing the true sceneInitialize to a volume V containing the true scene

Repeat until convergenceRepeat until convergence

Choose a voxel on the current surfaceChoose a voxel on the current surface

Carve if not photo-consistentCarve if not photo-consistentProject to visible input imagesProject to visible input images

Kutulakos & SeitzKutulakos & Seitz

Multi-Pass Plane SweepMulti-Pass Plane Sweep

• Faster alternative:Faster alternative:– Sweep plane in each of 6 principal directionsSweep plane in each of 6 principal directions

– Consider cameras on only one side of planeConsider cameras on only one side of plane

– Repeat until convergenceRepeat until convergence

Multi-Pass Plane SweepMulti-Pass Plane Sweep

True Scene Reconstruction

Multi-Pass Plane SweepMulti-Pass Plane Sweep

Multi-Pass Plane SweepMulti-Pass Plane Sweep

Multi-Pass Plane SweepMulti-Pass Plane Sweep

Multi-Pass Plane SweepMulti-Pass Plane Sweep

Multi-Pass Plane SweepMulti-Pass Plane Sweep

Space Carving Results: African Space Carving Results: African VioletViolet

Input Image (1 of 45) Input Image (1 of 45) ReconstructionReconstruction

ReconstructionReconstructionReconstructionReconstruction

Space Carving Results: HandSpace Carving Results: Hand

Input ImageInput Image(1 of 100) (1 of 100)

Views of ReconstructionViews of Reconstruction