Optimal Illumination for Image and Video Relighting

Francesc Moreno-Noguer Shree K. Nayar Peter N. Belhumeur

Department of Computer Science – Columbia University

European Conference on CVMP November 2005, London, UK

Objective

Objective

Objective

Objective

Relighting Static Objects

Relighting Static Objects

Relighting Static Objects

Relighting Static Objects

Relighting Static Objects

Relighting Static Objects

Relighting Static Objects

Relighting Static Objects

Relighting Static Objects

Relighting Static Objects

Relighting Static Objects

Relighting Static Objects

Relighting Static Objects

Relighting Static Objects

Relighting Static Objects

Relighting Static Objects

Relighting Static Objects

Relighting Static Objects

Relighting Static Objects

Relighting Static Objects

Relighting Static Objects

Relighting Static Objects

Relighting Static Objects

Relighting Static Objects

Relighting Static Objects

?

Relighting Static Objects

?

x x xx x x

x x x x x

x x x x x

x x x x x

x

x

x

Relighting Static Objects

x x xx x x

x x x x x

x x x x x

x x x x x

x

x

x

Relighting Static Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

?

Relighting Moving Objects

?

x x xx x x

x x x x x

x x x x x

x x x x x

x

x

x

Relighting Moving Objects

x x xx x x

x x x x x

x x x x x

x x x x x

x

x

x

Relighting Moving Objects

x x xx x x

x x x x x

x x x x x

x x x x x

x

x

x

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

Relighting Moving Objects

= I

= L

Relighting Moving Objects

= Ialign

= L

Relighting Moving Objects

Lnew

Relighting Moving Objects

= Ialign

= L

Lnew

?Inew

Relighting Moving Objects

= Ialign

= L

Relighting Moving Objects

Inew= Ialign pinv(L) Lnew

1 + 2 3 4+ +=

Inew= Ialign pinv(L) Lnew

Relighting Moving Objects

=

Relighting Moving Objects

Inew= Ialign pinv(L) Lnew

1 + 2 3 4+ +=

What is the optimal lighting?

What is the optimal lighting?

Lopt ?

Sources of error in video relighting

Sub-basis error

Ground Truth

Sources of error in video relighting

Sub-basis error

Ground Truth Rendered 3 basis

Error 3 basis

Sources of error in video relighting

Sub-basis error

Ground Truth Rendered 3 basis

Error 3 basis

Sources of error in video relighting

Sub-basis error

Ground Truth Rendered 3 basis

Error 3 basis

Rendered 6 basis

Error 6 basis

Sources of error in video relighting

Sub-basis error

Ground Truth Rendered 3 basis

Error 3 basis

Rendered 6 basis

Error 6 basis

Sources of error in video relighting

Alignment errorReference

frame

Sources of error in video relighting

Alignment errorReference

frameFrame

t+1

Sources of error in video relighting

Alignment errorReference

frame

Alignment errors

Frame t+1

Sources of error in video relighting

Alignment errorReference

frame

Alignment errors

Frame t+1

Frame t+2

Sources of error in video relighting

Alignment errorReference

frame

Alignment errors

Frame t+1

Frame t+2

Sources of error in video relighting

Alignment errorReference

frame

Alignment errors

Frame t+1

Frame t+2

Frame t+3

Sources of error in video relighting

Alignment errorReference

frame

Alignment errors

Frame t+1

Frame t+2

Frame t+3

Sources of error in video relighting

Alignment errorReference

frame

Alignment errors

Frame t+1

Frame t+2

Frame t+3

Frame t+4

Sources of error in video relighting

Alignment errorReference

frame

Alignment errors

Frame t+1

Frame t+2

Frame t+3

Frame t+4

Sources of error in video relighting

Alignment errorReference

frame

Alignment errors

Frame t+1

Frame t+2

Frame t+3

Frame t+4

Frame t+5

Sources of error in video relighting

Alignment errorReference

frame

Alignment errors

Frame t+1

Frame t+2

Frame t+3

Frame t+4

Frame t+5

Sources of error in video relighting

Orientation error

Sources of error in video relighting

Orientation error

Sources of error in video relighting

Reference frame

Orientation error

Sources of error in video relighting

Reference frame

Perfectly aligned frame

Orientation error

Sources of error in video relighting

Reference frame

Error afterperfect

alignment

Perfectly aligned frame

Orientation error

Sources of error in video relighting

Sub-basis error

Alignment error

Orientation error

Criterion for selecting the OLB

Minimize the sub-basis error

BUT

Keep the number of reference images to a minimum

Object independent lighting bases

Spherical Harmonic basis (SHLB)

Object independent lighting bases

Spherical Harmonic basis (SHLB)

Fourier basis (FLB)

Object independent lighting bases

Fourier basis (FLB)

Haar basis (HaLB)

Spherical Harmonic basis (SHLB)

Object dependent lighting basis

Optimal Lighting Basis (OLB)

• Computed as a simple calibration procedure before acquisition

• Initial acquisition of the images of the still object under single light sources

• Compute SVD over the matrix of images

• Computed as a simple calibration procedure before acquisition

• Initial acquisition of the images of the still object under single light sources

• Compute SVD over the matrix of images

...A=

A=UDWT

Object dependent lighting basis

A=UDWT

...A=

Object dependent lighting basis

A=UDWT

...A=

...U=

Object dependent lighting basis

A=UDWT

...A=

...

...

U=

W = T

Object dependent lighting basis

A=UDWT

...A=

...

...

...

U=

W = T

Lopt=

Object dependent lighting basis

Synthetic results

Optimal Lighting Basis (OLB)

Spherical Hamonic (SHLB)

Synthetic results

Optimal Lighting Basis (OLB)

Spherical Hamonic (SHLB)

Optimal Lighting Basis (OLB)

Fourier (FLB)

Haar (HaLB)

Synthetic results – Static objects

Ground Truth

Synthetic results – Static objects

Fourier 16 basis

Error

Fourier 16 basis

Ground Truth

Synthetic results – Static objects

Fourier 16 basis OLB 16 basis

Error

Fourier 16 basis

Error

OLB 16 basis

Ground Truth

Synthetic results – Static objects

Ground Truth

Synthetic results – Static objects

Haar 3 basis

Error

Haar 3 basis

Ground Truth

Synthetic results – Static objects

Haar 3 basis OLB 3 basis

Error

Haar 3 basis

Error

OLB 3 basis

Ground Truth

Synthetic results – Static objects

Ground Truth

Synthetic results – Static objects

Sph. Harm.

7 basis

Error

Sph. Harm. 7 basis

Ground Truth

Synthetic results – Static objects

Sph. Harm.

7 basis

OLB 7 basis

Error

Sph. Harm. 7 basis

Error

OLB 7 basis

Ground Truth

Synthetic results – Static objects

Ground Truth

Synthetic results – Static objects

Sph. Harm. 3 basis

Error

Sph. Harm. 3 basis

Ground Truth

Synthetic results – Static objects

Sph. Harm. 3 basis OLB 3 basis

Error

Sph. Harm. 3 basis

Error

OLB 3 basis

Ground Truth

Synthetic results – Static objects

Synthetic results – Static objects

Dragon Example:

Synthetic results – Static objects

8 OLB = 8 x 1.6 SHLB 13 SHLBDragon Example:

Synthetic results – Static objects

8 OLB = 8 x 1.6 SHLB 13 SHLB8 OLB = 8 x 1.9 SHLB 15 FLB

Dragon Example:

Synthetic results – Static objects

Dragon Example: 8 OLB = 8 x 1.6 SHLB 13 SHLB8 OLB = 8 x 1.9 SHLB 15 FLB8 OLB = 8 x 3.2 SHLB 25 HaLB

Frame #50

Synthetic results – Moving objects

Frame #1

OLB vs. SHLB

Frame #50

Synthetic results – Moving objects

Frame #1

9 OLB

OLB vs. SHLB

Frame #50

Synthetic results – Moving objects

Frame #1

9 OLB 16 SHLB

OLB vs. SHLB

Frame #50

Synthetic results – Moving objects

Frame #1

OLB vs. Fourier & Haar

Frame #50

Synthetic results – Moving objects

Frame #1

3 OLB

OLB vs. Fourier & Haar

Frame #50

Synthetic results – Moving objects

Frame #1

3 OLB 5 FLB

OLB vs. Fourier & Haar

Frame #50

Synthetic results – Moving objects

Frame #1

3 OLB 5 FLB

OLB vs. Fourier & Haar

16 HaLB

Real Experiments - Setup

Real Experiments - Procedure

OLB computation

1.1. Static scene, single lights1.2. SVD OLB

Video adquisition

2. Moving scene, OLB

Postprocessing 3.1. Alignment3.2. Relighting

Real Experiments – Tennis Ball

Real Experiments – Tennis Ball

Real Experiments – Face

Real Experiments – Face

Real Experiments – Face

Real Experiments – Room corner

Real Experiments – Room corner

Real Experiments – Room corner

Real Experiments – Room corner

Conclusions

• Proposed an “object-dependent” lighting basis appropriate for relighting tasks.

• Minimizes the number of reference images needed for relighting.

• Effective method for video relighting.

• Proposed an “object-dependent” lighting basis appropriate for relighting tasks.

• Minimizes the number of reference images needed for relighting.

• Effective method for video relighting.

Future work

• Use faster cameras.

• Use “motion-dependent” relighting bases.

• Use faster cameras.

• Use “motion-dependent” relighting bases.

Frame t+1

Frame t+2

Frame t+3

Frame t+4

Frame t+5

Frame t

Slow motion

More accurate results

Future work

• Use faster cameras.

• Use “motion-dependent” relighting bases.

• Use faster cameras.

• Use “motion-dependent” relighting bases.

Fast motion

Frame t+1

Frame t+2

Frame t+3

Frame t+4

Frame t+5

Frame t

Slow motion

Reduce alignment & orientation

errors

More accurate results

Acknowledgments

• Ko Nishino for his useful comments.

• Anne Fleming for her help in the edition of the videos.

• All the people from the “Laboratory for the Study of Visual Appearance” and “Columbia Automated Vision Environment's” at Columbia University.

• Cyberware for the 3D models.

• Funding Sponsors: • Spanish Ministry of Science & Technology• National Science Foundation

• Ko Nishino for his useful comments.

• Anne Fleming for her help in the edition of the videos.

• All the people from the “Laboratory for the Study of Visual Appearance” and “Columbia Automated Vision Environment's” at Columbia University.

• Cyberware for the 3D models.

• Funding Sponsors: • Spanish Ministry of Science & Technology• National Science Foundation