extracting objects from range and radiance images computer science division university of california...

Extracting Objects from Extracting Objects from Range and Radiance ImagesRange and Radiance Images

Extracting Objects from Extracting Objects from Range and Radiance ImagesRange and Radiance Images

Computer Science Division

University of California at Berkeley

Computer Science Division

University of California at Berkeley

Yizhou Yu Andras Ferencz Jitendra Malik

Image-based Modeling and RenderingImage-based Modeling and RenderingImage-based Modeling and RenderingImage-based Modeling and Rendering

– Vary viewpoint

– Vary lighting

– Vary scene configuration

– Vary viewpoint

– Vary lighting

– Vary scene configuration

Recover Models of Real World Scenes and Make Possible Various Visual Interactions


• 1st Generation---- vary viewpoint but not lighting– Recover geometry ( explicit or implicit )

– Acquire photographs

– Facade, Plenoptic Modeling, View Morphing, Lumigraph, Layered Depth Images, Concentric Mosaics, Light Field Rendering etc.

• 1st Generation---- vary viewpoint but not lighting– Recover geometry ( explicit or implicit )

– Acquire photographs

– Facade, Plenoptic Modeling, View Morphing, Lumigraph, Layered Depth Images, Concentric Mosaics, Light Field Rendering etc.


• 2nd Generation---- vary viewpoint and lighting– Recover geometry & reflectance properties

– Render using light transport simulation or local shading

– [ Yu, Debevec, Malik & Hawkins 99 ], [ Yu & Malik 98 ], [ Sato, Wheeler & Ikeuchi 97 ]

• 2nd Generation---- vary viewpoint and lighting– Recover geometry & reflectance properties

– Render using light transport simulation or local shading

– [ Yu, Debevec, Malik & Hawkins 99 ], [ Yu & Malik 98 ], [ Sato, Wheeler & Ikeuchi 97 ]

Original Lighting & Viewpoint Novel Lighting & Viewpoint


• 3rd Generation--Vary spatial configurations in addition to viewpoint and lighting

• 3rd Generation--Vary spatial configurations in addition to viewpoint and lighting

Novel Viewpoint Novel Viewpoint & Configuration

Our FrameworkOur FrameworkOur FrameworkOur Framework

• Input– Multiple range scans of a

scene

– Multiple photographs of the same scene

• Input– Multiple range scans of a

scene

– Multiple photographs of the same scene

• Output– Geometric meshes of each

object in the scene

– Registered texture maps for objects

• Output– Geometric meshes of each

object in the scene

– Registered texture maps for objects

Overview & VideoOverview & VideoOverview & VideoOverview & Video

RangeImages

RadianceImages

PointCloud

PointGroups

Meshes SimplifiedMeshes

CalibratedImages

TextureMaps Objects

Registration Segmentation Reconstruction

PoseEstimation

Texture MapSynthesis

OutlineOutlineOutlineOutline

• Overview

• Range Data Segmentation

• Radiance Image Registration

• Meshing and Texture-Mapping

• Results

• Future Work

• Overview

• Range Data Segmentation

• Radiance Image Registration

• Meshing and Texture-Mapping

• Results

• Future Work

OverviewOverviewOverviewOverview

RangeImages

RadianceImages

PointCloud

PointGroups


CalibratedImages

TextureMaps Objects


PoseEstimation


Range Image RegistrationRange Image RegistrationRange Image RegistrationRange Image Registration

• Registration with Calibration Objects– Cyra Technologies, Inc.

• Registration without Calibration Objects– [ Besl & McKay 92 ], [ Pulli 99 ]

• Registration with Calibration Objects– Cyra Technologies, Inc.

• Registration without Calibration Objects– [ Besl & McKay 92 ], [ Pulli 99 ]


RangeImages

RadianceImages

PointCloud

PointGroups


CalibratedImages

TextureMaps Objects


PoseEstimation


Previous Work on Range Image Previous Work on Range Image SegmentationSegmentationPrevious Work on Range Image Previous Work on Range Image SegmentationSegmentation

• Local region growing with surface primitives– [ Hoffman & Jain 87 ], [ Besl & Jain 88 ], [ Newman

Flynn & Jain 93 ], [ Leonardis, Gupta & Bajcsy 95 ], [ Hoover et. al. 96 ]

• They don’t address general free-form shapes and albedo variations.

• Local region growing is suboptimal in finding object boundaries.

• Local region growing with surface primitives– [ Hoffman & Jain 87 ], [ Besl & Jain 88 ], [ Newman

Flynn & Jain 93 ], [ Leonardis, Gupta & Bajcsy 95 ], [ Hoover et. al. 96 ]

• They don’t address general free-form shapes and albedo variations.

• Local region growing is suboptimal in finding object boundaries.

Normalized Cut FrameworkNormalized Cut FrameworkNormalized Cut FrameworkNormalized Cut Framework

),(

),(

),(

),(),(

VBasso

BAcut

VAasso

BAcutBANcut

DyyWD )(

[ Shi & Malik 97 ], [ Malik, Belongie, Shi & Leung 99 ]

Recursive Binary Graph Partition by Minimizing

Approximate Solution

From 2D Image To 3D Point CloudFrom 2D Image To 3D Point CloudFrom 2D Image To 3D Point CloudFrom 2D Image To 3D Point Cloud

• Neighborhood– 3D spherical region

• Complexity– E.g. nineteen 800 by 800 scans

– Clustering

• Neighborhood– 3D spherical region

• Complexity– E.g. nineteen 800 by 800 scans

– Clustering

Point Cloud Segmentation: CuesPoint Cloud Segmentation: CuesPoint Cloud Segmentation: CuesPoint Cloud Segmentation: Cues

Normal Orientation

Returned Laser Intensity

Proximity

EuuEvudist pnu /||||||||),( 22

)),(),,((MAX),( vudistvudistvudist vu

nu

puu

v

Point Cloud Segmentation: Graph SetupPoint Cloud Segmentation: Graph SetupPoint Cloud Segmentation: Graph SetupPoint Cloud Segmentation: Graph Setup

Nodes: Clusters

Edges: Local + Random Long-range Connections

Weights: ),(),(),( vuPvuSvuw

)2/),(diff(exp),( 22svuvuS

)2/),(dist(exp),( 22pvuvuP

u

v

),( vuw

Point Cloud Segmentation: CriterionPoint Cloud Segmentation: CriterionPoint Cloud Segmentation: CriterionPoint Cloud Segmentation: Criterion

• Use Normalized Cut Criterion to Propose Cut

• Use Normalized Weighted Average Cut to Accept Cut

• Use Normalized Cut Criterion to Propose Cut

• Use Normalized Weighted Average Cut to Accept Cut

)),(

),(,),(

),((),(

BBWAasso

BAWAcut

AAWAasso

BAWAcutMINBANWAcut

),(

),(),(),(

vuP

vuPvuSBAWAcut

Point Cloud Segmentation: AlgorithmPoint Cloud Segmentation: AlgorithmPoint Cloud Segmentation: AlgorithmPoint Cloud Segmentation: Algorithm

• Coarse Segmentation– Clustering

– Cluster Segmentation

• Recursive segmentation based on normal continuity and proximity

• Recursive segmentation based on continuity in laser intensity and proximity

• Fine Segmentation– re-clustering and segmentation on each group from

coarse segmentation

• Coarse Segmentation– Clustering

– Cluster Segmentation

• Recursive segmentation based on normal continuity and proximity

• Recursive segmentation based on continuity in laser intensity and proximity

• Fine Segmentation– re-clustering and segmentation on each group from

coarse segmentation

Segmentation ResultsSegmentation ResultsSegmentation ResultsSegmentation Results


RangeImages

RadianceImages

PointCloud

PointGroups


CalibratedImages

TextureMaps Objects


PoseEstimation


Previous Work on Pose EstimationPrevious Work on Pose EstimationPrevious Work on Pose EstimationPrevious Work on Pose Estimation

• Mathematical Background– from 3 or more points [Hung et.al. ‘85],

[Haralick et.al. ‘89], Overview [Haralick et.al. ‘94]

– from points, lines and ellipse-circle pairs [Qiang et.al. ‘99]

• Automatic Techniques– Combinatorial search on automatically detected

features [Huttenlocher and Ulman ‘90],[Wunsch and Hirzinger ‘96]

– Hybrid approach (user initialization with finding silhouettes) [Neugebauer and Klein ‘99]

• Mathematical Background– from 3 or more points [Hung et.al. ‘85],

[Haralick et.al. ‘89], Overview [Haralick et.al. ‘94]

– from points, lines and ellipse-circle pairs [Qiang et.al. ‘99]

• Automatic Techniques– Combinatorial search on automatically detected

features [Huttenlocher and Ulman ‘90],[Wunsch and Hirzinger ‘96]

– Hybrid approach (user initialization with finding silhouettes) [Neugebauer and Klein ‘99]

Radiance Image Registration Radiance Image Registration Radiance Image Registration Radiance Image Registration

• Requirements– Automatic: many (50 - 200) images need to be

registered

– Accurate: Registration must be accurate to within one or two pixels for texture mapping

– General purpose: scene may be complicated, possibly without easily and uniquely identifiable features

• Solution– Place registration targets in scene

• Requirements– Automatic: many (50 - 200) images need to be

registered

– Accurate: Registration must be accurate to within one or two pixels for texture mapping

– General purpose: scene may be complicated, possibly without easily and uniquely identifiable features

• Solution– Place registration targets in scene

Finding Targets in ImagesFinding Targets in ImagesFinding Targets in ImagesFinding Targets in Images

• Pattern matching followed by ellipse fitting• Pattern matching followed by ellipse fitting

Combinatorial SearchCombinatorial SearchCombinatorial SearchCombinatorial Search• Brute force

– 4 correspondences each image, O(n4) time n targets

• Alternative– Use fitted ellipse parameters in addition to position to

estimate pose from 2 target matches, O(n2) time

• 20 seconds for scene with 100 targets

• Brute force– 4 correspondences each image, O(n4) time n targets

• Alternative– Use fitted ellipse parameters in addition to position to

estimate pose from 2 target matches, O(n2) time

• 20 seconds for scene with 100 targets

3D Targets

Registration SummaryRegistration SummaryRegistration SummaryRegistration Summary

• Place registration targets in scene before acquiring data

• Automatically detect targets in data

• Perform combinatorial search to match image targets to corresponding target in geometry

• Find camera pose from matched points

• Remove targets from images and fill in holes

• Place registration targets in scene before acquiring data

• Automatically detect targets in data

• Perform combinatorial search to match image targets to corresponding target in geometry

• Find camera pose from matched points

• Remove targets from images and fill in holes

Camera Pose ResultsCamera Pose ResultsCamera Pose ResultsCamera Pose Results

• Accuracy: consistently within 2 pixels

• Correctness: correct pose for 58 out of 62 images

• Accuracy: consistently within 2 pixels

• Correctness: correct pose for 58 out of 62 images


RangeImages

RadianceImages

PointCloud

PointGroups


CalibratedImages

TextureMaps Objects


PoseEstimation


Mesh Reconstruction and SimplificationMesh Reconstruction and SimplificationMesh Reconstruction and SimplificationMesh Reconstruction and Simplification

• Meshing– Use the “crust” algorithm, [ Amenta, Bern &

Kamvysselis 98 ], for coarse geometry

– Use nearest-neighbor connections for detailed geometry

– Possible to use volumetric techniques, [ Curless & Levoy 96 ], to merge meshes

• Simplification– Use quadric error metric, [ Garland & Heckbert 97 ]

• Meshing– Use the “crust” algorithm, [ Amenta, Bern &

Kamvysselis 98 ], for coarse geometry

– Use nearest-neighbor connections for detailed geometry

– Possible to use volumetric techniques, [ Curless & Levoy 96 ], to merge meshes

• Simplification– Use quadric error metric, [ Garland & Heckbert 97 ]

Reconstructed Mesh with Camera Poses Reconstructed Mesh with Camera Poses and Calibration Targetsand Calibration TargetsReconstructed Mesh with Camera Poses Reconstructed Mesh with Camera Poses and Calibration Targetsand Calibration Targets

Models of Individual ObjectsModels of Individual ObjectsModels of Individual ObjectsModels of Individual Objects


RangeImages

RadianceImages

PointCloud

PointGroups


CalibratedImages

TextureMaps Objects


PoseEstimation


Texture Map SynthesisTexture Map SynthesisTexture Map SynthesisTexture Map Synthesis

• Regular Texture-Mapping with Texture Coordinates– Compose a triangular texture

patch for each triangle.

– Allocate space from texture maps for each texture patch to assign texture coordinates.

– The size of each texture patch is determined by the amount of variations in photographs.

– The derivative of the Gaussian is used as a metric for variations.

• Regular Texture-Mapping with Texture Coordinates– Compose a triangular texture

patch for each triangle.

– Allocate space from texture maps for each texture patch to assign texture coordinates.

– The size of each texture patch is determined by the amount of variations in photographs.

– The derivative of the Gaussian is used as a metric for variations.

Photograph

Texture Map

3D Triangle

Texture-Mapping and Object Texture-Mapping and Object ManipulationManipulationTexture-Mapping and Object Texture-Mapping and Object ManipulationManipulation

VideoVideoVideoVideo

Future WorkFuture WorkFuture WorkFuture Work

• Create watertight geometric models from sparse or incomplete data

• Improve mesh simplification techniques

• Texture Compression

• Create watertight geometric models from sparse or incomplete data

• Improve mesh simplification techniques

• Texture Compression

AcknowledgmentsAcknowledgmentsAcknowledgmentsAcknowledgments

• Thanks to Ben Kacyra, Mark Wheeler, Daniel Chudak, Jonathan Kung at Cyra Technologies, Inc., and Jianbo Shi at CMU.

• Supported by ONR BMDO, the California MICRO program, and Microsoft Graduate Fellowship.

• Thanks to Ben Kacyra, Mark Wheeler, Daniel Chudak, Jonathan Kung at Cyra Technologies, Inc., and Jianbo Shi at CMU.

• Supported by ONR BMDO, the California MICRO program, and Microsoft Graduate Fellowship.

extracting objects from range and radiance images computer science division university of california...

Documents