pixelwise view selection for unstructured multi-view stereo · •joint depth - normal - occlusion...

Pixelwise View Selection for Unstructured Multi-View StereoJohannes L. Schönberger1 Enliang Zheng2 Marc Pollefeys1,3 Jan-Michael Frahm2

1ETH Zürich 2UNC Chapel Hill 3Microsoft

Overview

Joint Depth - Normal - Occlusion InferenceThis work presents an open source Multi-View Stereo system for robust and efficient dense modeling from unstructured image collections. Experiments on benchmarks and large-scale Internet photo collections demonstrate state-of-the-art performance in terms of accuracy, completeness, and efficiency.

Contributions

• Joint depth - normal - occlusion inferenceembedded in improved PatchMatch sampling scheme

• Pixelwise view selectionusing photometric and geometric priors

• Multi-view geometric consistencyfor simultaneous refinement and image-based fusion

• Graph-based filtering and fusionof depth and normal maps

Multi-View Geometric Consistency

Pixelwise View Selection

• Occlusion prior

• Triangulation prior

• Resolution prior

• Incident prior

Ref. patch Source patches

Reference camera

Source camera

Source camera

-1 -0.5 0 0.5 1

NCC score

0

0.5

1

Occ

lusi

on P

rior

σρ = 0.3

σρ = 0.6

σρ = 0.9

0 50 100 150

Incident angle [deg]

0

0.5

1

Inci

dent

Prio

r

σκ = 15◦

σκ = 30◦

σκ = 45◦

0 5 10 15 20 25 30

Triangulation angle [deg]

0

0.5

1

Tria

ngul

atio

n P

rior

α = 2◦

α = 5◦

α = 10◦

α = 15◦

0 1 2 3 4 5

Relative resolution blm with b

l = 1

0

0.5

1

Res

olut

ion

Prio

r

Source patchesRef. patch

Traditional Pixelwise

a

S

Pix

• Joint likelihood function

• Generalized Expectation Maximization (GEM) •E-Step: Infer using variational inference •M-Step: Infer using PatchMatch sampling

Z θ,N

ξml = 1−ρml +ηmin (ψml , ψmax)

Photometric Geometric

Cost function

Optimization argminθ∗

l,n∗

l

1

|S|

∑m∈S

ξml (θ∗l ,n∗

l )

Filtering and Fusion

ψml

Filtering

Fusion

ResultsZheng et al. Photometric Photometric + Geometric Filtered Normals

https://colmap.github.ioSource Code | Documentation | Tutorial | Examples

P (αml ) = 1−

(min(α,αm

l)−α)2

α2

P (βml ) = min(βm

l , (βml )−1)

P (κml ) = exp(−

κm

l

2

2σ2κ

)

P (Xml |Zm

l , θl) =

{1

NAexp

(−

(1−ρm

l(θl))

2

2σ2ρ

)if Zm

l = 1

1

NU if Zm

l = 0

P (X,Z, θ,N)

NormalsDepthOcclusionImages

P (αml )

P (βml )

P (κml )

P (Xml |Zm

l , θl)