deriving intrinsic images from image sequences mohit gupta 04/21/2006 advanced perception yair weiss
TRANSCRIPT
![Page 1: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/1.jpg)
Deriving Intrinsic Images from Image Sequences
Mohit Gupta
04/21/2006
Advanced Perception
Yair Weiss
![Page 2: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/2.jpg)
Intrinsic Scene Characteristics• Introduced by Barrow and Tanenbaum, 1978
• Motivation: Early visual system decomposes image into ‘intrinsic’ properties
Input Image Reflectance Orientation Illumination Distance
![Page 3: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/3.jpg)
Intrinsic Images
Input = Reflectance x Illumination
• Mid-Level description of scenes
• Information about intrinsic scene properties
• Falls short of a full 3D description
![Page 4: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/4.jpg)
Motivation
• Information about scene properties: prior for visual inference tasks
Segmentation: Invariant to illumination
Original Illumination
Reflectance
![Page 5: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/5.jpg)
Motivation
• Information about scene properties: prior for visual inference tasks
Shape from Shading: Invariant to reflectance
Original Illumination
Reflectance
![Page 6: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/6.jpg)
Problem Definition• Given I, solve for L and R such that
I(x,y) = L(x,y) * R(x,y)
I = Input ImageL = Illumination ImageR = Reflectance Image
![Page 7: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/7.jpg)
Problem Definition• Given I, solve for L and R such that
I(x,y) = L(x,y) * R(x,y)
(disturbed ) This is preposterous!!
You can’t possibly solve this !!
Dr. Math
Classical Ill Posed Problem:
# Unknowns = 2 * # Equations
![Page 8: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/8.jpg)
Problem Definition• Given I, solve for L and R such that
I(x,y) = L(x,y) * R(x,y)
(disturbed ) This is preposterous!!
You can’t possibly solve this !!
Dr. Math
Classical Ill Posed Problem:
# Unknowns = 2 * # Equations
Hey doc, Don’t PANIC
These pixels ‘hang out together’ a lot
Mohit
Exploit ‘structure’ in the images to reduce the no. of
unknowns !
![Page 9: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/9.jpg)
Previous Work Retinex Algorithm [Land and McCann]
Reflectance image piecewise constant
Illumination is attached shadows (photometric sterero)
L(x,y,t) = N(x,y) . S(t)
Illumination images related by a scalar L(x,y,t) = (t) * L(x,y)
![Page 10: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/10.jpg)
Previous Work Retinex Algorithm [Land and McCann]
Reflectance image piecewise constant
Illumination is attached shadows (photometric sterero)
L(x,y,t) = N(x,y) * S(t)
Illumination images related by a scalar L(x,y,t) = (t) * L(x,y)
All exploit temporal or spatial structure
in the images to reduce the no. of unknowns !
![Page 11: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/11.jpg)
Cut to the present…
R(x,y,t) = R(x,y)
•Motivation
• Lot of web-cam images
• Stationary camera, reflectance doesn’t change
•This paper relies on temporal structure
![Page 12: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/12.jpg)
Cut to the present…
R(x,y,t) = R(x,y)
•Motivation
• Lot of web-cam images
• Stationary camera, reflectance doesn’t change
•This paper relies on temporal structure
I(x,y,t) = R(x,y) * L(x,y,t)
T equations, T+1 unknowns
Still an Ill-Posed Problem !!
![Page 13: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/13.jpg)
Slight Detour:Background Extraction
Problem: Given a sequence of images I(x,y,t), extract the stationary component, or the ‘background’ from them
Images:
Alyosha Efros
![Page 14: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/14.jpg)
Image Stack
t0
255time
We can look at the set of images as a spatio-temporal volume Each line through time corresponds to a single pixel in
space If camera is stationary, we can decompose the image
as:
image static background dynamic foreground
i(x,y,t) = b(x,y) + f(x,y,t)Images:
Alyosha Efros
![Page 15: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/15.jpg)
Power of Median Image
image static background dynamic foreground
i(x,y,t) = b(x,y) + f(x,y,t)
Key Observation: If for each pixel (x,y), f(x,y,t) = 0 ‘most of the times’
then
b(x,y) = mediant i(x,y,t)
Example: b(x,y) = 42; f(x,y,t) = [0, 2, 3, 0, 0]; i(x,y,t) = [42, 44, 45, 42, 42]
b(x,y) = median( [42,44,45,42,42]) = 42 !
![Page 16: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/16.jpg)
Power of Median Image
![Page 17: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/17.jpg)
Power of Median Image
Median Image =
Background !
![Page 18: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/18.jpg)
Background Extraction & Intrinsic Images
I(x,y,t) = L(x,y,t) * R(x,y)i(x,y,t) = l(x,y,t) + r(x,y) (log)
Compare to i(x,y,t) = f(x,y,t) + b(x,y)
Static Background = Reflection ImageMoving Foregrounds = Illumination Images
(shadows)
Intrinsic Image Equation
![Page 19: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/19.jpg)
Trouble!Illumination Images, l(x,y,t) sparse: Not a safe
assumption
Median Image “Shady” Result
![Page 20: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/20.jpg)
Key Idea: Lets look at gradient images…
Gradients of shadows are sparse, even though the shadows aren’t !
Rationale: Smoothness of shadows
![Page 21: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/21.jpg)
Key Idea: Lets look at gradient images…
Gradients of shadows are sparse, even though the shadows aren’t !
Rationale: Smoothness of shadowsi(x,y,t) = l(x,y,t) + r(x,y) gradient if(x,y,t) = lf(x,y,t) + rf(x,y)
![Page 22: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/22.jpg)
Key Idea: Lets look at gradient images…
Gradients of shadows are sparse, even though the shadows aren’t !
Rationale: Smoothness of shadowsi(x,y,t) = l(x,y,t) + r(x,y) gradient if(x,y,t) = lf(x,y,t) + rf(x,y)
lf(x,y,t) is sparse
rf(x,y) = mediant if(x,y,t)
![Page 23: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/23.jpg)
Median Gradient Image
Filtered Reflectance image
rf(x,y) = mediant if(x,y,t)
Recovered Reflectance image
![Page 24: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/24.jpg)
Median Gradient Image
Filtered Reflectance image Recovered Reflectance image
![Page 25: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/25.jpg)
Median Gradient Image
Filtered Reflectance image Recovered Reflectance image
I(x,y,t) = R(x,y) * L(x,y,t)
T equations, T+1 unknowns
Still an Ill-Posed Problem ?
No, sparsity of gradient illumination images
imposes additional constraints!
![Page 26: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/26.jpg)
Recovering image from Gradient Images
f(x,y)Horizontal filtered image (v1)
Vertical filtered image (v2)
f = v
f = . v
(del operator)
Poisson Equation: f = g (from gradient images: g = .v)
Along with the boundary condition
v = (v1,v2)
![Page 27: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/27.jpg)
Recovering image from Gradient Images
f(x,y)Horizontal filtered image (v1)
Vertical filtered image (v2)
f = v
f = . v
(del operator)
Poisson Equation: f = g (from gradient images: g = .v)
Along with the boundary coundition
v = (v1,v2)
Interpretation of solving the Poisson equation: Computes the function (f) whose
gradient is the closest to the guidance vector field (v), under given boundary conditions.
![Page 28: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/28.jpg)
Recovering image from Gradient Images
f(x,y)Horizontal filtered image (v1)
Vertical filtered image (v2)
f = v
f = . v
(del operator)
Poisson Equation: f = g (from gradient images: g = .v)
v = (v1,v2)
Boundary can be from mean of input images – hope that edges are mostly shadow-free
+
![Page 29: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/29.jpg)
Poisson Image Editing (Perez, Gangnet, Blake, SIGGRAPH ’03)
Source Destination
Cloning Poisson Blendin
g
Want to find a new function f, which ‘looks like’ g in the interior and like
f* near the boundary
Use g as guiding vector field with f* providing the boundary condition
![Page 30: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/30.jpg)
Poisson Image Editing (Perez, Gangnet, Blake, SIGGRAPH ’03)
![Page 31: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/31.jpg)
The Algorithm
1. Filter outputs for input image (on) are calculated
2. Filtered reflectance image (rn) is computed as rn(x,y) = mediant on (x,y,t)
3. Reflectance image r is recovered from rn
4. Illumination images are recovered using the relation: l(x,y,t) = i(x,y,t) – r(x,y)
![Page 32: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/32.jpg)
Results : Synthetic
frame i frame j ML illumination
(frame i)
ML reflectance
** Note that the pixels surrounding the diamond are always in shadow, yet their estimated reflectance is the same as that of pixels that were always in light.
![Page 33: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/33.jpg)
Results : Real World
![Page 34: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/34.jpg)
Results : Real World
![Page 35: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/35.jpg)
Some fun …
Original Image Logo belnded with Image
Logo blended with reflectance image, and
rendered with corresponding illumination
image
![Page 36: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/36.jpg)
Limitations
• Requires multiple images of a static scene in different lighting
• Highly sensitive to input - scene content and sequence length (basically a shadow detector !)
• Can't remove static shadows
• High complexity - filtering the images and finding median are high cost functions.
![Page 37: Deriving Intrinsic Images from Image Sequences Mohit Gupta 04/21/2006 Advanced Perception Yair Weiss](https://reader036.vdocuments.mx/reader036/viewer/2022081519/56649ef45503460f94c07af9/html5/thumbnails/37.jpg)
Conclusions• Fully automatic algorithm to derive intrinsic images from a sequence of images
• Simplification by making constant reflectance assumption
• Use sparsity of gradient images to derive a simple solution
• Paper has a rather complex statistical derivation for the same result !
• Doesn’t tackle the original problem of recovering intrinsic images from a single image ( next presentation)