shadow removal algorithms
DESCRIPTION
Shadow removal algorithms. Shadow removal seminar Pavel Knur. Deriving intrinsic images from image sequences. Yair Weiss July 2001. History. “ intrinsic images ” by Barrow and Tenenbaum , 1978. Constraints. Fixed viewpoint Works only for static objects Cast shadows. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/1.jpg)
Shadow removal algorithms
Shadow removal seminarPavel Knur
![Page 2: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/2.jpg)
Deriving intrinsic images from image sequences
Yair WeissJuly 2001.
![Page 3: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/3.jpg)
History
• “intrinsic images” by Barrow and Tenenbaum , 1978
![Page 4: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/4.jpg)
Constraints
• Fixed viewpoint• Works only for static objects• Cast shadows
![Page 5: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/5.jpg)
Classic ill-posed problem
•Denote– the input image– the reflectance image– the illumination image
Number of Unknowns is twice the number of equations.
),( yxR),( yxI
),( yxL
),(),(),( yxRyxLyxI
![Page 6: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/6.jpg)
The problem
Given a sequence of T imagesin which reflectance is constant over the time and only the illuminationchanges, can we solve for a singlereflectance image and T illumination images ?
Still completely ill-posed : at every pixel there are T equations and T+1 unknowns.
)},,({1
tyxIT
t
)},,({1
tyxLT
t
![Page 7: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/7.jpg)
Maximum-likelihood estimation
• Log domain :
),(),(
),(),(
),(),(
log
log
log
yxlyxL
yxryxR
yxiyxI
),,(),(),,( tyxlyxrtyxi
![Page 8: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/8.jpg)
Assumptions
When derivative filters are applied to natural images, the filter outputs tend to be sparse.
![Page 9: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/9.jpg)
Laplacian distribution
Can be well fit by laplacian distribution
xZ exP 1)(
![Page 10: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/10.jpg)
Claim 1
Denote :• N filters – • Filter outputs – • Filtered reflectance image –
ML estimation of filtered reflectance image
is given by
}{ nf
nn ftyxityxo ),,(),,(
nn frr
nr̂
),,(ˆ tyxomedianr ntn
![Page 11: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/11.jpg)
Estimated reflectance function
Recover ML estimation of r
is reversed filter of
nn rrf ˆˆ
)ˆ(ˆ n
nrn rfgr
rnf nf
)(n
nrn ffg
![Page 12: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/12.jpg)
ML estimation algorithm
![Page 13: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/13.jpg)
ML estimation algorithm – cont.
• Ones we have estimated ),( yxr
),(),,(),,( yxrtyxityxl
![Page 14: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/14.jpg)
Claim 2
•What if does not have exactly a Laplasian distribution ?
Let
Then estimated filtered reflectance are within with probability at least:
),,( tyxlfn
)),,(( tyxlfPp i
2/
1
)1(T
k
kkT ppk
T
![Page 15: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/15.jpg)
Claim 2 - proof
If more than 50% of the samples ofare within of some value, then by definition of median, the median must be within of that value.
),,( tyxlfn
![Page 16: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/16.jpg)
Example 1
• Einstein image is translated diagonally
• 4 pixels per frame
![Page 17: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/17.jpg)
Example 2
• 64 images with variable lighting from Yale Face Database
![Page 18: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/18.jpg)
Illumination Normalization with Time-Dependent Intrinsic Images for Video SurveillanceY.Matsushita,K.Nishito,K.IkeuchiOct. 2004
![Page 19: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/19.jpg)
Illumination Normalization algorithm
• Preprocessing stage for robust video surveillance.
• Causes– Illumination conditions– Weather conditions– Large buildings and trees
• Goal– To “normalize” the input image
sequence in terms of incident lighting.
![Page 20: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/20.jpg)
Constraints
• Fixed viewpoint• Works only for static objects• Cast shadows
![Page 21: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/21.jpg)
Background images
• Remove moving objects from the input image sequence
Input images
Background images
Off-line
![Page 22: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/22.jpg)
Estimation of Intrinsic Images
Denote• input image• time-varying reflectance image• time-varying illumination image• reflectance image estimated by ML• illumination image estimated by
ML
• Filters
• Log domain
Input images
Background images
Off-line
Estimation of Intrinsic Images
),,( tyxR
),,( tyxL
),( yxRw),,( tyxLw
),,( tyxI
),,(),,(),,( tyxRtyxLtyxI
1100 f
Tf 1101
ww lrlri ,,,,
![Page 23: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/23.jpg)
Estimation of Intrinsic Images – cont.
• In Weiss’s original work
• The goal is to find estimation of and
Input images
Background images
Off-line
Estimation of Intrinsic Images
),,(),(ˆ tyxifmedianyxr ntwn
),(ˆ),,(),,(ˆ yxrtyxiftyxl wnnwn
ril
),,( tyxR ),,( tyxL
![Page 24: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/24.jpg)
Estimation of Intrinsic Images – cont.
Basic idea:• Estimate time-varying reflectance
components by canceling the scene texture from initial illumination images
Define:
Input images
Background images
Off-line
Estimation of Intrinsic Images
otherwisetyxl
Tyxriftyxl
wn
wnn ),,,(
),(,0),,(
otherwiseyxr
Tyxriftyxlyxrtyxr
wn
wnwnwnn ),,(
),(),,,(),(),,(
),,(),,(),,(),(),,( tyxltyxrtyxlyxrtyxif nnwnwnn
![Page 25: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/25.jpg)
Estimation of Intrinsic Images – cont.
Finally :
Where : is reversed filter of
Input images
Background images
Off-line
Estimation of Intrinsic Images
nn
rn
nn
rn
lfgtyxl
rfgtyxr
ˆ),,(ˆ
ˆ),,(ˆ
r
nf nf
)(n
nrn ffg
![Page 26: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/26.jpg)
Shadow Removal
Denote - background image - illuminance-invariant image
Input images
Background images
Off-line
Estimation of Intrinsic Images
),,( tyxB
),,(),,(),,( tyxLtyxRtyxB
),,( tyxN
),,(/),,(),,( tyxLtyxBtyxN
![Page 27: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/27.jpg)
Illumination Eigenspace
• PCA – Principle component analysisBasic components -
Input images
Background images
Off-line
Estimation of Intrinsic Images
nsss ,...,, 21
IlluminationEigenspace
![Page 28: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/28.jpg)
Illumination Eigenspace – cont.
• Average is
• P is MxN matrix where– N – number of pixels in illumination
image– M – number of illumination images
• Covariance matrix Q of P is
Input images
Background images
Off-line
Estimation of Intrinsic Images
n ww L
nL
1
IlluminationEigenspace
wwwwww LLLLLLPn ,...,,
21
TPPQ
iii Qee
![Page 29: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/29.jpg)
Direct Estimation of Illumination Images
• Pseudoillumination image
• Direct Estimation is
• Where– F is a projection function onto the j’s
eigenvector
-
Input images
Background images
Off-line
Estimation of Intrinsic Images
),(/),,(* yxRtyxIL w
IlluminationEigenspace
j
wjLw jLFjLFwLiiw
2* ),(),(minargˆ
i
jjw
![Page 30: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/30.jpg)
Direct Estimation of Illumination Images
• Results
Input images
Background images
Off-line
Estimation of Intrinsic Images
IlluminationEigenspace
![Page 31: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/31.jpg)
Shadow interpolation
probability density functioncumulative probability functionshadowed arealit area
mean
optimum threshold value
Input images
Background images
Off-line
Estimation of Intrinsic Images
IlluminationEigenspace
ShadowInterpolation
T
iis ipTP
min
)()(
max
)()(i
Til ipTP
T
iis iipT
min
)()(
max
)()(i
Til iipT
2)()()()(maxarg TTTPTPT lsls
T
opt
)(ip
Ps
l
optT
![Page 32: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/32.jpg)
The whole algorithmInput images
Background images
Off-line
Estimation of Intrinsic Images
IlluminationEigenspace
/
IlluminationImages
Normalization
ShadowInterpolation
![Page 33: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/33.jpg)
Example
![Page 34: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/34.jpg)
Questions ?
![Page 35: Shadow removal algorithms](https://reader035.vdocuments.mx/reader035/viewer/2022062518/56814aad550346895db7c188/html5/thumbnails/35.jpg)
References
[1] Y.Weiss,”Deriving Intrinsic Images from Image Sequences”, Proc. Ninth IEEE Int’l Conf. Computer Vision, pp. 68-75, July 2001.
[2] Y.Matsushita,K.Nishito,K.Ikeuchi,“Illumination Normalization with Time-Dependent Intrinsic Images for Video Surveillance”,Oct. 2004.