Francesc Moreno-NoguerComputer Vision Lab.Ecole Polytechnique Fédérale de Lausanne

Peter N. BelhumeurShree K. NayarColumbia University

SIGGRAPH 2007

Active Refocusing of Images and Videos

study

Abstract

Use an active illumination method for depth estimation from a single image

Acquired ImageComputed Depth

NearFar

Refocused (Near)Refocused (Far)Alternate Lighting

Outlines

Introduction Related Work Overview Projection Dot Defocus Analysis Dot Removal & Depth Estimation Realistic Refocusing Result Limits and Conclusions

Introduction of Refocusing

Challenges of Active Refocusing Dynamic scenes

Depth Estimation be done in a single frame

Active illumination Full resolution depth map Projection Dot removal

Partial Occlusions

xfy k captured

blur kernels at depth k

In-focus

RELATED WORK

Relative Work: Depth EstimationPassive Methods

Active Illumination Methods

Shape from shading Cannot handle depth

discontinuities

Coded Aperture [Levin et al. SIGGRAPH 07] Cam. H.W. modify Require Light Source

Pattern

Structured Light [Salvi et al. Pattern Recognition ,04] No pattern removal

Projector Temporal Defocus [Zhang & Nayar SIGGRAPH06]

Relative Work: Digital RefocusingRefocusing Given Depth

Light Field Photography

Synthesis Images: Ray Tracing [Cook SIGGRAPH84] Require complete 3D model

Real Images: Convolution [Photoshop; IrisFilter] Partial Occlusions Problem

Light Field Camera [Ng SIGGRAPH05] Cam. H.W. modify Resolution losses

Dappled Photography [Veeraraghavan SIGGRAPH07] Cam. H.W. modify Layer

Sparse Depth Map

Acquired Image

Dots Removed

Color Segmentation

Merged Segmentation

Dense Depth

Dots Depth Estimation by

Calibration

Dots Removal

Matting

Depth Map Completion using Segmentation

Depth Estimation

Depth Map

Realistic Refocusing

Dots Removed

Focal plane,Apertures,Window size of dots

PROJECTION DOT DEFOCUS ANALYSIS

System Design

Camera & Projector Coaxial have same Optical Axis

Projector

Blur Circle Diameter, D

fc uurfD

112

uf

Dr

u

fcv

v

fwDD c

w

w

with dot size w (in the projector plane)

Blur Circle Radiance, I

21I

f

p

uu

uf

Dr

u

fcv

2

1

2

I

vr

wu

u

uw

f

w

w

with dot size w (in the projector plane)

based on Image Irradiance Equation derived in [Horn 86]

Camera images of dot of 3*3 pixels projected onto different depths

Camera images of dot of 3*3 pixels projected onto different depths

DOT REMOVAL AND DEPTH ESTIMATION

Calibration Patches

Estimated

Sparse Depth MapDepth 1Depth 2 …

X =

Calibration Patches

Estimated

Sparse Depth MapDepth 1Depth 2 …

Calibration Patches

Estimated

Sparse Depth MapDepth 1Depth 2 …

Depth Estimation - ux

Non-textured Surface

Textured Surfaces (texture by itself introduces brightness variation)

riN

j ijcic ,, II

riN

jijc

iix uu ,Ivarminarg|

ici

ix uu ,Iminarg|

based on Unsupervised Learning Alg. [Figueiredo and Jain IEEE02]

DEPTH MAP COMPLETION USING SEGMENTATION

Depth Map Completion

Sparse Depth Map

Over-Segmentation

Mean-Shift[Comaniciu & Meer 02]

Iterative Merging

Depth Map Completion – Iterative Merging

Loop: Apply Greedy alg. to group segments Merge the two most

similar neighboring segments

Re-computes the features of the new merged segment

Iterative Merging

Similarity between Segments

Sim(i,j)=λC∙dist(Ci,Cj)+λD∙dist(Di,Dj)+λT∙dist(Ti,Tj)

Color C Depth D Texture T

Depth Map Completion – Refine the Depth Disc.

Noisy Depth Map

Matting Algorithm[Wang & Cohen

05]

REALISTIC REFOCUSING

Challenge of Refocusing Partial occlusions

Different parts of the lens may see different views at an object boundary

Create missing region by detecting discontinuities in depth map and extending the occluded surface using texture synthesis

Foreground/background transitions Pixels at depth discontinuities may

receive contributions from the fr. and bg. Blend fr./bg. images within the

boundary region

A)(RRR 1A GCFC

)A-(1RARR GCFC

Realistic Refocusing produces better results than existing approaches

Original

Realistic Refocusin

g

Canon + wide

aperture

Photoshop - blur

IrisFilter

Partial Occlusions

Refocusing with Alpha Maps

R RC Є F* += *R

C Є B

Background (B)

Boundary (C)

Foreground (F)

RC Є F

RC Є B

RESULT

Limitations

Due to Active Illumination

Translucent objects exhibit subsurface scattering

Blurred dots are too weak to detect Very dark Highly inclined surface

(> 70°)

Poor in outdoor with strong sunlight

ex: the ball and the table are assigned diff. depths due to errors on segmentation errors

Limitations

Due to sparse dots Sparsity of the depth estimation

Errors in the initial segmentation of the image

ex: incorrect depth due to segmentation err.

Conclusions

Contribution Future Work

An active illumination depth estimation with single Single Frame, Complete

Depth Map, Texture/Textureless scenes

Projected Light Patterns are Removed

High resolution refocusing of images and videos

Incorporate the method into digital cameras

Use intra-red source for projecting the dot patter to make the depth estimation more robust in the case of highly textured scenes

END