20100822 computervision boykov
TRANSCRIPT
![Page 1: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/1.jpg)
Energy Minimization with Label Costsand Model Fitting
presented by Yuri Boykov
co-authors:Andrew Delong Anton Osokin Hossam Isack
![Page 2: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/2.jpg)
The University of
OntarioOverview Standard models in vision (focus on discrete case)
• MRF/CRF, weak-membrane, discontinuity-preserving...• Information-based: MDL (Zhu&Yuille’96) , AIC/BIC (Li’07)
Label costs and their optimization • LP-relaxations, heuristics, α-expansion++
Model Fitting• dealing with infinite number of labels ( PEARL )
Applications• unsupervised image segmentation• geometric model fitting (lines, circles, planes, homographies, ...)• rigid motion estimation• extensions…
![Page 3: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/3.jpg)
The University of
Ontario
Reconstruction in Vision: (a basic example)
L
observed noisy image I image labeling L(restored intensities)
How to compute L from I ?
I
L = { L1, L2 , ... , Ln }I = { I1, I2 , ... , In }
![Page 4: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/4.jpg)
The University of
Ontario
Energy minimization(discrete approach)
MRF framework• weak membrane model (Geman&Geman’84, Blake&Zisserman’83,87)
pL qL
ZZLp 2:
Nqp
qpp
pp LLVILE),(
2 ),()()(L ,V
T Tdiscontinuity preserving potentials
Blake&Zisserman’83,87
spatial regularizationdata fidelity
![Page 5: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/5.jpg)
The University of
OntarioOptimization Convex regularization
• gradient descent works• exact polynomial algorithms
TV regularization• a bit harder (non-differentiable)• global minima algorithms (Ishikawa, Hochbaum, Nikolova et al.)
Robust regularization• NP-hard, many local minima• good approximations (message passing, a-expansion)
,V
,V
,V
Nqp
qpp
pp LLVILE),(
2 ),()()(L
![Page 6: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/6.jpg)
The University of
Ontario
Potts model(piece-wise constant labeling)
Robust regularization• NP-hard, many local minima• provably good approximations (a-expansion)
,V
)(),( TwV
maxflow/mincut combinatorial algorithms
Nqp
qpp
pp LLVILE),(
2 ),()()(L
![Page 7: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/7.jpg)
The University of
Ontario
Left eye imageRight eye image
Potts model(piece-wise constant labeling)
Robust regularization• NP-hard, many local minima• provably good approximations (a-expansion)
,V
)(),( TwV
depth layers
maxflow/mincut combinatorial algorithms
Nqp
qpp
pp LLVLDE),(
),()()(L
![Page 8: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/8.jpg)
The University of
Ontario
Potts model(piece-wise constant labeling)
Robust regularization• NP-hard, many local minima• provably good approximations (a-expansion)
,V
)(),( TwV
0
C
1
maxflow/mincut combinatorial algorithms
Nqp
qpp
pp LLVLDE),(
),()()(L
![Page 9: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/9.jpg)
The University of
OntarioAdding label costs
Lippert [PAMI 89]• MDL framework, annealing
Zhu and Yuille [PAMI 96]• continuous formulation (gradient des cent )
H. Li [CVPR 2007]• AIC/BIC framework, only 1st and 3rd terms• LP relaxation (no guarantees approximation)
Our new work [CVPR 2010] , extended a-expansion • all 3 terms, 3rd term is represented as some high-order clique), optimality bound• very fast heuristics for 1st & 3rd term (facility location problem, 60-es)
L
LLN)q,p(
qpp
pp )(h)L,L(V)L(D)(E LL
- set of labelsallowed at each
point p
otherwise
LLp pL ,0
:,1)(L
![Page 10: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/10.jpg)
The University of
OntarioThe rest of the talk…
Why label costs?
![Page 11: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/11.jpg)
The University of
OntarioModel fitting
pL
||Lp||minargL̂
}b,a{L
p
2xy )bapp(||Lp||
y=ax+b
SSD
![Page 12: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/12.jpg)
The University of
Ontariomany outliersquadratic errors fail
use more robust error measures, e.g.
gives “MEDIAN” line|bapp|||Lp|| xy
- more expensive computations
(non-differentiable)- still fails if
outliers exceed 50%
RANSAC
![Page 13: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/13.jpg)
The University of
Ontariomany outliers
RANSAC
1. sample randomlytwo points, get a line
![Page 14: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/14.jpg)
The University of
Ontariomany outliers
10 inliers
RANSAC
1. sample randomlytwo points, get a line2. count inliers for
threshold T
![Page 15: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/15.jpg)
The University of
Ontariomany outliers
30 inliers
1. sample randomlytwo points, get a line2. count inliers for
threshold T
3. repeat N times and select
model with most inliers
RANSAC
![Page 16: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/16.jpg)
The University of
OntarioMultiple models and many outliers
Why not RANSAC
again?
![Page 17: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/17.jpg)
The University of
OntarioMultiple models and many outliers
In general, maximization of inliersdoes not work for
outliers + multiple models
Why not RANSAC
again?
Higher noise
![Page 18: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/18.jpg)
The University of
OntarioEnergy-based approach
p
||Lp||)L(E
energy-based interpretation of RANSAC criteria forsingle model fitting:
- find optimal label Lfor one very specific
error measure
TdistifTdistif
dist,1,0
||||
![Page 19: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/19.jpg)
The University of
OntarioEnergy-based approach
Npq
qpp
p LLTwLpE )(||||)(L
If multiple models
- assign different models (labels Lp) to every point
p
- find optimal labelingL = { L1, L2 , ... , Ln }
Need regularization!
![Page 20: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/20.jpg)
The University of
OntarioEnergy-based approach
Npq
qpp
p LLTwLpE )(||||)(L
If multiple models
- assign different models (labels Lp) to every point
p
- find optimal labelingL = { L1, L2 , ... , Ln }
![Page 21: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/21.jpg)
The University of
OntarioEnergy-based approach
If multiple models
- assign different models (labels Lp) to every point
p
- find optimal labelingL = { L1, L2 , ... , Ln }
L
LLp
p )(h||Lp||)(E LL
otherwise
LLp pL ,0
:,1)(L
- set of labelsallowed at each
point p
![Page 22: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/22.jpg)
The University of
OntarioEnergy-based approach
If multiple models
- assign different models (labels Lp) to every point
p
- find optimal labelingL = { L1, L2 , ... , Ln }
L
LLN)q,p(
qpp
p )(h)LL(Tw||Lp||)(E LL
Practical problem: number of potential labels (models) is huge, how are we going to use a-expansion?
![Page 23: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/23.jpg)
The University of
OntarioPEARL
data points
ProposeExpandAndReestimateLabels
![Page 24: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/24.jpg)
The University of
OntarioPEARL
data points + randomly sampled models
sample datato generatea finite set
of initial labels
ProposeExpandAndReestimateLabels
![Page 25: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/25.jpg)
The University of
OntarioPEARL
models and inliers (labeling L)
a-expansion:minimize E(L)
segmentationfor fixed
set of labels
Npq
qpp
p )LL(Tw||Lp||)(E L
ProposeExpandAndReestimateLabels
![Page 26: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/26.jpg)
The University of
OntarioPEARL
ProposeExpandAndReestimateLabels
models and inliers (labeling L)
reestimating labels in
for given inliers
minimizes first term
of energy E(L)
Npq
qpp
p )LL(Tw||Lp||)(E L
![Page 27: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/27.jpg)
The University of
OntarioPEARL
ProposeExpandAndReestimateLabels
models and inliers (labeling L)
Npq
qpp
p )LL(Tw||Lp||)(E L
a-expansion:minimize E(L)
segmentationfor fixed
set of labels
![Page 28: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/28.jpg)
The University of
OntarioPEARL
iterate until convergence
Npq
qpp
p )LL(Tw||Lp||)(E L
after 5 iterations
ProposeExpandAndReestimateLabels
![Page 29: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/29.jpg)
The University of
Ontario
PEARL can significantlyimprove initial models
single line fittingwith 80% outliers
number of initial samplesde
viat
ion
(fr
om g
roun
d tr
uth
)
![Page 30: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/30.jpg)
The University of
Ontario
Comparison formulti-model fitting
original data points
Low noise
![Page 31: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/31.jpg)
The University of
Ontario
Comparison formulti-model fitting
some generalization of RANSAC
Low noise
![Page 32: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/32.jpg)
The University of
Ontario
Comparison formulti-model fitting
PEARL
Low noise
![Page 33: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/33.jpg)
The University of
Ontario
Comparison formulti-model fitting
original data points
High noise
![Page 34: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/34.jpg)
The University of
Ontario
Comparison formulti-model fitting
Some generalization of RANSAC (Multi-RANSAC, Zuliani et al. ICIP’05)
High noise
![Page 35: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/35.jpg)
The University of
Ontario
Comparison formulti-model fitting
Other generalization of RANSAC (J-linkage, Toldo & Fusiello, ECCV’08)
High noise
![Page 36: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/36.jpg)
The University of
Ontario
Comparison formulti-model fitting
Finding modes in Hough-space, e.g. via mean-shift(also maximizes the number of inliers)
Hough transform
High noise
![Page 37: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/37.jpg)
The University of
Ontario
Comparison formulti-model fitting
PEARL
High noise
![Page 39: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/39.jpg)
The University of
OntarioFitting circles
Here spatial regularization does not work well
regularization with label costs only
![Page 40: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/40.jpg)
The University of
OntarioFitting planes (homographies)
Original image (one of 2 views)
![Page 41: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/41.jpg)
The University of
OntarioFitting planes (homographies)
(a) Label costs only
![Page 42: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/42.jpg)
The University of
OntarioFitting planes (homographies)
(b) Spatial regularity only
![Page 43: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/43.jpg)
The University of
OntarioFitting planes (homographies)
(c) Spatial regularity + label costs
![Page 44: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/44.jpg)
The University of
Ontario
(unsupervised) Image Segmentation
Original image
![Page 45: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/45.jpg)
The University of
Ontario
(unsupervised) Image Segmentation
(a) Label costs only [Li, CVPR 2007]
![Page 46: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/46.jpg)
The University of
Ontario
(unsupervised) Image Segmentation
(b) Spatial regularity only [Zabih&Kolmogorov CVPR 04]
![Page 47: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/47.jpg)
The University of
Ontario
(unsupervised) Image Segmentation
(c) Spatial regularity + label costs
Zhu and Yuille 96used continuous
variational formulation(gradient discent)
![Page 48: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/48.jpg)
The University of
Ontario
(unsupervised) Image Segmentation
(c) Spatial regularity + label costs
![Page 49: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/49.jpg)
The University of
Ontario
(unsupervised) Image Segmentation
Spatial regularity + label costs
![Page 50: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/50.jpg)
The University of
Ontario
(unsupervised) Image Segmentation
Spatial regularity + label costs
![Page 51: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/51.jpg)
The University of
Ontario
(unsupervised) Image Segmentation
Spatial regularity + label costs
![Page 52: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/52.jpg)
The University of
Ontario
(unsupervised) Image Segmentation
Spatial regularity + label costs
![Page 53: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/53.jpg)
The University of
Ontario
(rigid)Motion Estimation
Original image
3 motions
[Rene Vidal]
![Page 54: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/54.jpg)
The University of
Ontario
(rigid)Motion Estimation
(a) Label costs only
3 motions
![Page 55: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/55.jpg)
The University of
Ontario
(rigid)Motion Estimation
(b) Spatial regularity only
7 motions
![Page 56: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/56.jpg)
The University of
Ontario
(rigid)Motion Estimation
(c) Spatial regularity + label costs
3 motions
![Page 61: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/61.jpg)
The University of
OntarioPlane fitting
Note very small steps between each floor
![Page 62: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/62.jpg)
The University of
Ontario
Affine model fitting(from a rectified stereo pair)
photoconsistency + smoothness
dense model assignments to pixels
Birchfield & Tomasi’99(fit initial models to output of other stereo
algorithm ++ α-expansion + reestimation)
geometric errors + smoothness+ label cost
sparse model assignments to features
PEARL(sample data + α-expansion + reestimation)
![Page 63: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/63.jpg)
The University of
Ontario
Duh...use right geometric error measure!!!
“disparity” errors d1 and d2 (bad idea!)“quatient”-based errors d (standard)
![Page 64: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/64.jpg)
The University of
Ontario
Affine model fitting(from a rectified stereo pair)
photoconsistency + smoothness
dense model assignments to pixels
Birchfield & Tomasi’99(fit initial models to output of other stereo
algorithm ++ α-expansion + reestimation)
geometric errors + smoothness+ label cost
sparse model assignments to features
PEARL(sample data + α-expansion + reestimation)
![Page 65: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/65.jpg)
The University of
Ontario
Photoconsistency vs. Geometric Alignment
photoconsistency optimization (Birchfield & Tomasi’99)
densestereo
![Page 66: 20100822 computervision boykov](https://reader031.vdocuments.mx/reader031/viewer/2022022205/58d0033c1a28abfc0a8b695f/html5/thumbnails/66.jpg)
The University of
Ontario
Photoconsistency vs. Geometric Alignment
geometric error minimization via PEARL
sparse data
sparsestereo