![Page 1: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/1.jpg)
6.801/866
Segmentation and Line Fitting
T. Darrell
![Page 2: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/2.jpg)
Segmentation and Line Fitting
• Gestalt grouping• Background subtraction• K-Means• Graph cuts• Hough transform• Iterative fitting
(Next time: Probabilistic segmentation)
![Page 3: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/3.jpg)
Segmentation and Grouping
• Motivation: vision is often simple inference, but for segmentation
• Obtain a compact representation from an image/motion sequence/set of tokens
• Should support application• Broad theory is absent at
present
• Grouping (or clustering)– collect together tokens that
“belong together”
• Fitting– associate a model with
tokens– issues
• which model?• which token goes to which
element?• how many elements in the
model?
![Page 4: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/4.jpg)
General ideas
• tokens– whatever we need to group
(pixels, points, surface elements, etc., etc.)
• top down segmentation– tokens belong together
because they lie on the same object
• bottom up segmentation– tokens belong together
because they are locally coherent
• These two are not mutually exclusive
![Page 5: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/5.jpg)
Why do these tokens belong together?
![Page 6: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/6.jpg)
What is the figure?
![Page 7: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/7.jpg)
Basic ideas of grouping in humans
• Figure-ground discrimination– grouping can be seen in
terms of allocating some elements to a figure, some to ground
– impoverished theory
• Gestalt properties– elements in a collection of
elements can have properties that result from relationships (Muller-Lyer effect)
• gestaltqualitat– A series of factors affect
whether elements should be grouped together
• Gestalt factors
![Page 8: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/8.jpg)
![Page 9: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/9.jpg)
![Page 10: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/10.jpg)
![Page 11: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/11.jpg)
![Page 12: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/12.jpg)
Occlusion is an important cue in grouping.
![Page 13: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/13.jpg)
![Page 14: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/14.jpg)
![Page 15: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/15.jpg)
Technique: Background Subtraction
• If we know what the background looks like, it is easy to identify “interesting bits”
• Applications– Person in an office– Tracking cars on a road– surveillance
• Approach:– use a moving average to
estimate background image– subtract from current frame– large absolute values are
interesting pixels• trick: use morphological
operations to clean up pixels
![Page 16: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/16.jpg)
![Page 17: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/17.jpg)
low thresh high thresh
EM (later)
80x60
![Page 18: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/18.jpg)
low thresh high thresh
EM (later)
160x120
![Page 19: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/19.jpg)
Classic Background Subtraction model
• Background is assumed to be mostly static• Each pixel is modeled as by a gaussian
distribution in YUV space• Model mean is usually updated using a recursive
low-pass filter
Given new image, generate silhouetteby marking those pixels that are significantlydifferent from the “background” value.
![Page 20: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/20.jpg)
Finding Features
2D Head / hands localization– contour analysis: mark extremal points (highest
curvature or distance from center of body) as hand features
– use skin color model when region of hand or face is found (color model is independent of flesh tone intensity)
![Page 21: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/21.jpg)
Static Background Modeling Examples
[MIT Media Lab Pfinder / ALIVE System]
![Page 22: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/22.jpg)
Static Background Modeling Examples
[MIT Media Lab Pfinder / ALIVE System]
![Page 23: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/23.jpg)
Static Background Modeling Examples
[MIT Media Lab Pfinder / ALIVE System]
![Page 24: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/24.jpg)
Dynamic Background
BG Pixel distribution is non-stationary:
[MIT AI Lab VSAM]
![Page 25: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/25.jpg)
Mixture of Gaussian BG model
Staufer and Grimson tracker:Fit per-pixel mixture model to observed distrubution.
[MIT AI Lab VSAM]
![Page 26: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/26.jpg)
Segmentation as clustering
• Cluster together (pixels, tokens, etc.) that belong together
• Agglomerative clustering– attach closest to cluster it is
closest to– repeat
• Divisive clustering– split cluster along best
boundary– repeat
• Point-Cluster distance– single-link clustering– complete-link clustering– group-average clustering
• Dendrograms– yield a picture of output as
clustering process continues
![Page 27: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/27.jpg)
Clustering Algorithms
![Page 28: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/28.jpg)
![Page 29: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/29.jpg)
K-Means
• Choose a fixed number of clusters
• Choose cluster centers and point-cluster allocations to minimize error
• can’t do this by search, because there are too many possible allocations.
• Algorithm– fix cluster centers; allocate
points to closest cluster– fix allocation; compute best
cluster centers
• x could be any set of features for which we can compute a distance (careful about scaling)
x j − µ i
2
j∈elements of i'th cluster∑
i∈clusters
∑
![Page 30: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/30.jpg)
K-Means
![Page 31: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/31.jpg)
Image Clusters on intensity (K=5) Clusters on color (K=5)
K-means clustering using intensity alone and color alone
![Page 32: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/32.jpg)
Image Clusters on color
K-means using color alone, 11 segments
![Page 33: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/33.jpg)
K-means usingcolor alone,11 segments.
Oversegmentation!
![Page 34: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/34.jpg)
K-means using colour andposition, 20 segments
Still misses goal of perceptuallypleasing segmentation!
![Page 35: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/35.jpg)
Graph theoretic clustering
• Avoid local minima; use global criteria
• Represent tokens using a weighted graph.– affinity matrix
• Cut up this graph to get subgraphs with strong interior links
![Page 36: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/36.jpg)
Image Segmentation as Graph Partitioning
Build a weighted graph G=(V,E) from image
V: image pixels
E: connections between pairs of nearby pixels
region same the tobelong
j& iy that probabilit :ijW
Partition graph so that similarity within group is large and similarity between groups is small -- Normalized Cuts[Shi&Malik 97]
[Malik]
![Page 37: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/37.jpg)
Some Terminology for Graph Partitioning
• How do we bipartition a graph:
disjointy necessarilnot A' andA
A'A,
),(W)A'A,( ∑∈∈
=vu
vuassoc
∅=∩
∈∈∑=
BAwith
BA,
),,W(B)A,(vu
vucut
[Malik]
![Page 38: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/38.jpg)
![Page 39: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/39.jpg)
![Page 40: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/40.jpg)
Boundaries of image regions defined by a number of attributes
– Brightness/color– Texture– Motion– Stereoscopic depth– Familiar configuration
[Malik]
![Page 41: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/41.jpg)
Measuring Affinity
Intensityaff x, y( )= exp − 1
2σ i2
I x( )− I y( ) 2( )
Distance
aff x, y( )= exp − 12σ d
2
x − y 2( )
Color
aff x, y( )= exp − 12σ t
2
c x( )− c y( ) 2( )
![Page 42: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/42.jpg)
Eigenvectors and cuts
• Simplest idea: we want a vector a giving the association between each element and a cluster
• We want elements within this cluster to, on the whole, have strong affinity with one another
• We could maximize
• But need the constraint
• This is an eigenvalueproblem - choose the eigenvector of A with largest eigenvalue
aT Aa
aTa = 1
![Page 43: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/43.jpg)
Example eigenvector
eigenvector
points
matrix
![Page 44: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/44.jpg)
Scale affects affinity
σ=.2
σ=.1 σ=.2 σ=1
![Page 45: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/45.jpg)
Scale affects affinity
σ=.2
σ=.1 σ=.2 σ=1
![Page 46: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/46.jpg)
More than two segments
• Two options– Recursively split each side to get a tree, continuing till
the eigenvalues are too small– Use the other eigenvectors
![Page 47: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/47.jpg)
More than two segments
![Page 48: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/48.jpg)
Normalized cuts
• Current criterion evaluates within cluster similarity, but not across cluster difference
• Instead, we’d like to maximize the within cluster similarity compared to the across cluster difference
• Write graph as V, one cluster as A and the other as B
• Maximize
where cut(A,B) is sum of weights that straddle A,B; assoc(A,V) is sum of all edges with one end in A.
I.e. construct A, B such that their within cluster similarity is high compared to their association with the rest of the graph
cut(A,B)assoc(A,V)
cut(A,B)assoc(B,V)
+
![Page 49: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/49.jpg)
Normalized Cut
• Minimum cut (total weight of edges removed) is not appropriate since it favors cutting small pieces.
• Normalized Cut, Ncut:
V),(B)A,(
V)A,(B)A,( B)A,(
Bassoccut
assoccutNcut +=
[Malik]
![Page 50: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/50.jpg)
Solving the Normalized Cut problem
• Exact discrete solution to Ncut is NP-complete even on regular grid,– [Papadimitriou’97]
• Drawing on spectral graph theory, good approximation can be obtained by solving a generalized eigenvalue problem.
[Malik]
![Page 51: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/51.jpg)
Normalized cuts
• Write a vector y whose elements are 1 if item is in A, -b if it’s in B
• Write the matrix of the graph as W, and the matrix which has the row sums of W on its diagonal as D, 1 is the vector with all ones.
• Criterion becomes
and we have a constraint
• This is hard to solve, because y’s values are quantized
minyyT D −W( )y
yT Dy
yT D1 = 0
∑=i
ijii WD
![Page 52: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/52.jpg)
Normalized Cut As Generalized Eigenvalue problem
...
),(
),( ;
11)1()1)(()1(
11)1)(()1(
)VB,()BA,(
)VA,(B)A,(B)A,(
0
=
=−
−−−+
+−+=
+=
∑∑ >
i
xT
T
T
T
iiD
iiDk
DkxWDx
DkxWDx
assoccut
assoccutNcut
i
• after simplification, we get
.01},,1{ with ,)(),( =−∈−
= DybyDyy
yWDyBANcut TiT
T
[Malik]
![Page 53: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/53.jpg)
Normalized cuts
• Instead, solve the generalized eigenvalue problem
• which gives
• Now look for a quantization threshold that maximizes the criterion --- i.e all components of y above that threshold go to one, all below go to -b
maxy yT D − W( )y( ) subject to yT Dy = 1( )
D − W( )y = λDy
![Page 54: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/54.jpg)
(using intensity and texture affinity)Figure from “Image and video segmentation: the normalised cut framework”, by Shi and Malik, copyright IEEE, 1998
![Page 55: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/55.jpg)
(using motion / spatio-temporal affinity)
F igure from “Normalized cuts and image segmentation,” Shi and Malik, copyright IEEE, 2000
![Page 56: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/56.jpg)
Fitting
• Choose a parametric object/some objects to represent a set of tokens
• Most interesting case is when criterion is not local– can’t tell whether a set of
points lies on a line by looking only at each point and the next.
• Three main questions:– what object represents this
set of tokens best?– which of several objects
gets which token?– how many objects are
there?
(you could read line for object here, or circle, or ellipse or...)
![Page 57: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/57.jpg)
Fitting and the Hough Transform
• Purports to answer all three questions– in practice, answer isn’t
usually all that much help
• We do for lines only• A line is the set of points
(x, y) such that
• Different choices of θ, d>0 give different lines
• For any (x, y) there is a one parameter family of lines through this point, given by
• Each point gets to vote for each line in the family; if there is a line that has lots of votes, that should be the line passing through the points
sinθ( )x + cosθ( )y + d = 0sinθ( )x + cosθ( )y + d = 0
![Page 58: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/58.jpg)
tokensvotes
![Page 59: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/59.jpg)
Mechanics of the Hough transform
• Construct an array representing θ, d
• For each point, render the curve (θ, d) into this array, adding one at each cell
• Difficulties– how big should the cells be?
(too big, and we cannot distinguish between quite different lines; too small, and noise causes lines to be missed)
• How many lines?– count the peaks in the
Hough array
• Who belongs to which line?– tag the votes
• Hardly ever satisfactory in practice, because problems with noise and cell size defeat it
![Page 60: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/60.jpg)
tokens votes
![Page 61: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/61.jpg)
![Page 62: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/62.jpg)
![Page 63: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/63.jpg)
![Page 64: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/64.jpg)
Line fitting
What criteria to optimize when fitting a line to a set of points?
![Page 65: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/65.jpg)
“Least Squares”
“Total Least Squares”
Line fitting can be max.likelihood - but choice ofmodel is important
![Page 66: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/66.jpg)
Who came from which line?
• Assume we know how many lines there are - but which lines are they?– easy, if we know who came from which line
• Three strategies– Incremental line fitting– K-means– Probabilistic (later!)
![Page 67: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/67.jpg)
![Page 68: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/68.jpg)
Incremental line fitting
![Page 69: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/69.jpg)
Incremental line fitting
![Page 70: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/70.jpg)
Incremental line fitting
![Page 71: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/71.jpg)
Incremental line fitting
![Page 72: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/72.jpg)
Incremental line fitting
![Page 73: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/73.jpg)
![Page 74: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/74.jpg)
K-means line fitting
![Page 75: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/75.jpg)
K-means line fitting
![Page 76: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/76.jpg)
K-means line fitting
![Page 77: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/77.jpg)
K-means line fitting
![Page 78: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/78.jpg)
K-means line fitting
![Page 79: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/79.jpg)
K-means line fitting
![Page 80: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/80.jpg)
K-means line fitting
![Page 81: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/81.jpg)
Robustness
• As we have seen, squared error can be a source of bias in the presence of noise points– One fix is EM - we’ll do this shortly– Another is an M-estimator
• Square nearby, threshold far away
– A third is RANSAC• Search for good points
(Next lecture….)
![Page 82: Segmentation and Line Fitting T. Darrellcs.haifa.ac.il/~dkeren/acv/mit19.pdfSegmentation and Line Fitting • Gestalt grouping • Background subtraction •K-Means • Graph cuts](https://reader030.vdocuments.mx/reader030/viewer/2022041110/5f0f12c37e708231d4425b7d/html5/thumbnails/82.jpg)
Segmentation and Line Fitting
• Gestalt grouping• Background subtraction• K-Means• Graph cuts• Hough transform• Iterative fitting
(Next time: Probabilistic segmentation)
[Most figures from F&P or http://iram.cs.berkeley.edu/RetreatJul2000/malik.ppt]