segmentation (continued) - umartalhaumartalha.weebly.com/.../8382263/05_lecture_cv_nust_pnec.pdf ·...

Segmentation (continued)

Lecture 05 Computer Vision

• Dr George Stockman

Professor Emeritus, Michigan State University

• Dr Mubarak Shah

Professor, University of Central Florida

• The Robotics Institute

Carnegie Mellon University

• Dr David A. Forsyth

Professor, University of Illinois at Urbana- Champaign

• Dr Kristen Grauman

Associate Professor, University of Texas at Austin

Material Citations

Suggested Readings

• Chapter 10

Linda G. Shapiro and George Stockman, “Computer Vision”, Upper Saddle River, NJ, Prentice Hall, 2001 .

• Chapter 15

David A. Forsyth and Jean Ponce, “Computer Vision A Modern Approach”, 2nd edition, Prentice Hall, Inc., 2003.

• Chapter 3

Mubarak Shah, “Fundamentals of Computer Vision”.

Clustering

Main idea: Cluster together (pixels, tokens, etc.) that belong together Tokens: Whatever we need to group (pixels, points, surface elements, etc.)

Segmentation by Clustering

R3

R1 R2

R5

R4

R6


Main idea: Cluster together (pixels, tokens, etc.) that belong together Simple Methods • Divisive clustering

– Split cluster along best boundary – Repeat

• Agglomerative clustering – Attach token to cluster it is closest to – Repeat

Inter-Cluster distance criteria • Single-link clustering

– Distance between closest elements of clusters • Complete-link clustering

– Maximum distance between elements of clusters • Group-average clustering

– Average distance between elements of clusters


Dendrograms – A representation of structure of hierarchy of clusters

Dataset Dendogram

Segmentation by Clustering – K Means

Objective function • Assuming we know there are k clusters and k is known • Each cluster has center • Center of ith cluster is ci • The jth element to be clustered is described by feature vector xj (could be any set of features) • The objective function can be given as:

x j i2

jelements of i'th cluster

iclusters

Φ (clusters, data) = Properties • Will always converge to some solution • Can be a “local minimum”; doesn't always find the “global minimum”

function that it is desired to maximize or minimize.

The k-means problem is finding the least-squares assignment to centroids.


Desirables • Choose a fixed number of clusters • Choose cluster centers • Point-cluster allocations to minimize error

Algorithm • Fix cluster centers; allocate points to closest • Fix allocation; compute best cluster centers

Andrew Moore



• Select a feature vector for every pixel (color, texture, position, or combination of these etc.) • Define a similarity measure between feature vectors (Usually Euclidean Distance) • Apply K-Means Algorithm • Apply Connected Components Algorithm • Merge any components of size less than some threshold to an adjacent component that is most similar to it

K-means clustering using intensity alone and color alone

Image Clusters on intensity Clusters on color


No requirement that clusters be spatially localized and they’re not

Example - Segmenting

How to determine the three main intensities?

Kristen Grauman, University of Texas at Austin

Kristen Grauman, University of Texas at Austin

Depending on what we choose as the feature space, we can group pixels in different ways.

Grouping pixels based on intensity similarity

Feature space: intensity value (1-d)


K=2

K=3

quantization of the feature space; segmentation label map



R=255 G=200 B=250

R=245 G=220 B=248

R=15 G=189 B=2

R=3 G=12 B=2

R

G

B

Grouping pixels based on color similarity

Feature space: color value (3-d)

Kristen Grauman



Grouping pixels based on intensity similarity

Clusters based on intensity similarity don’t have to be spatially coherent.


Kristen Grauman

X

Y

Intensity

Both regions are black, but if we also include position (x,y), then we could group the two into distinct segments; way to encode both similarity & proximity.



Grouping pixels based on intensity + position similarity

Kristen Grauman

• Color, brightness, position alone are not enough to distinguish all regions…


Kristen Grauman

Feature space: filter bank responses (e.g., 24-d)

Filter bank

of 24 filters

F24

F2

F1

…



Grouping pixels based on texture similarity

Kristen Grauman

Texture representation example

Windows with small gradient in both directions

Both

Dimension 1 (mean d/dx value)

Dim

en

sio

n 2

(m

ean

d/d

y va

lue

)

Windows with primarily vertical edges

Windows with primarily horizontal edges

Kristen Grauman


Segmentation with texture features

• Find “textons” by clustering vectors of filter bank outputs

• Describe texture in a window based on texton histogram

Mal ik, Belongie, Leung and Shi. IJCV 2001. Adapted from Lana Lazebnik

Texton map Image

Texton index Texton index

Cou

nt

Cou

nt

Co

un

t

Texton index


Kristen Grauman



Pros • Simple, fast to compute • Converges to local minimum of within-cluster squared error Cons/issues • Setting k? • Sensitive to initial centers • Sensitive to outliers • Detects spherical clusters • Assuming means can be computed

Sample Code

http://www.mathworks.com/matlabcentral/fileexchange/8379-kmeans-image-segmentation

http://www.mathworks.com/matlabcentral/fileexchange/8379-kmeans-image-segmentationhttp://www.mathworks.com/matlabcentral/fileexchange/8379-kmeans-image-segmentationhttp://www.mathworks.com/matlabcentral/fileexchange/8379-kmeans-image-segmentationhttp://www.mathworks.com/matlabcentral/fileexchange/8379-kmeans-image-segmentationhttp://www.mathworks.com/matlabcentral/fileexchange/8379-kmeans-image-segmentationhttp://www.mathworks.com/matlabcentral/fileexchange/8379-kmeans-image-segmentationhttp://www.mathworks.com/matlabcentral/fileexchange/8379-kmeans-image-segmentation

• The mean shift algorithm seeks modes or local maxima of density in the feature space

image Feature space

(L*u*v* color values)

Segmentation by Clustering – Mean Shift

• Mean Shift Algorithm

– Choose a search window size.

– Choose the initial location of the search window.

– Compute the mean location (centroid of the data) in the search window.

– Center the search window at the mean location computed in Step 3.

– Repeat Steps 3 and 4 until convergence.


Search window

Center of mass

Mean Shift vector


Slide by Y. Ukrainitz & B. Sarel

Search window

Center of mass



• Cluster: all data points in the attraction basin of a mode

• Attraction basin: the region for which all trajectories lead to the same mode



• Mean Shift Segmentation Algorithm

– Convert the image into tokens (via color, gradients, texture measures etc).

– Choose initial search window locations uniformly in the data.

– Compute the mean shift window location for each initial position.

– Merge windows that end up on the same “peak” or mode.

– The data these merged windows traversed are clustered together.


• Find features (color, gradients, texture, etc)

• Initialize windows at individual feature points

• Perform mean shift for each window until convergence

• Merge windows that end up near the same “peak” or mode


Szeliski Book

http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html


• Pros:

– Does not assume shape on clusters

– One parameter choice (window size)

– Generic technique

– Find multiple modes

• Cons:

– Selection of window size

– Does not scale well with dimension of feature space

Kristen Grauman


Segmentation by Graph theoretic clustering

Graph Theory • A graph is a non-empty finite set of vertices V along with a set E of 2- element subsets of V. • The elements of V are called Vertices. • The elements of E are called edges.


• Represent tokens using A weighted undirected graph G = (V,E) • Nodes are points in the feature space • Fully connected graph • The weight associated are generally called affinity measures and Edge weight w(i,j) is a function of the similarity between nodes i and j. • The graphs can be represented by weight or affinity matrices

Task: • Cut up this graph to get sub-graphs with strong interior links, i.e. Partition the set V into disjoint sets V1,..,Vn, s.t. similarity among nodes in Vi is high and similarity across Vi and Vj is low.

Segmentation by Graph Cuts

• Break Graph into Segments

– Want to delete links that cross between segments

– Easiest to break links that have low similarity (low weight)

• Similar pixels should be in the same segments

• Dissimilar pixels should be in different segments

w

Source: Steve Seitz

A B C

q

p

wpq


Graph Representations

a a

b

b c

c

d

d

e

e

V = {a,b,c,d,e}

E = {(a,b), (a,e), (c,e), (d,e)}

Cardinality: No of vertices |G| = 5

Degree of vertex: Nodes connecting to a vertex deg(a) = 2

Adjacency Matrix

Weighted Graphs and Their Representations

a

a

b

b c

c

d

d

e

e

Measuring Affinity

Distance

Intensity

aff x, y exp 12 i

2

I x I y

2

aff x, y exp 12 d

2

x y

2

aff x, y exp 12 t

2

c x c y

2

Texture

Minimum Cut

A cut of a graph G is the set of edges S such that removal of S from G disconnects G Given a graph G=(V,E), the sets A and B are a disjoint partition of V Cut(A,B) is a measure of similarity between the two groups. Cut(A,B) is the sum of weights of all edges in V that have one end in A and the other in B Minimum cut is the cut of minimum weight, where weight of Cut is given as There can be more than one minimum cut in a given graph

A B

,

,

( , ) p qp A q B

Cut A B w

Minimum Cut

• Problem with minimum cut:

Weight of cut proportional to number of edges in the cut; tends to produce small, isolated components.

Minimum Cut

Ideal Cut

Cuts with lesser weight than the ideal cut

Normalized Cut

• We’d like to maximize the within cluster similarity compared to the across cluster difference • In other words the approach is to cut the graph into two connected components such that the cost of the cut is a small fraction of the total affinity within each group

Normalized Cut

• Normalized cut is defined as:

( , ) ( , )

( , ) ( , )

Cut A B Cut A B

Assoc A V Assoc B V

i.e. construct A, B such that their within cluster similarity is high compared to their association with the rest of the graph

Cut(A,B) = Sum of weights of all edges in V that touch both A and B Assoc(A,V) = Sum of weights of all edges that touch A

Ncut value small when we get two clusters with many edges with high weights, and few edges of low weight between them Approximate solution for minimizing the Ncut value is Generalized eigenvalue problem

det( ) 0A I

Finding Minimum Normalized Cut

Normalized Cut - Example results

Berkley Segmentation Engine http://www.cs.berkeley.edu/~fowlkes/BSE/

http://www.cs.berkeley.edu/~fowlkes/BSE/

Pros:

• Generic framework, flexible to choice of function that computes weights (“affinities”) between nodes

• Does not require model of the data distribution

Cons:

• Time complexity can be high

• Bias towards partitioning into equal segments

Sample student project:

http://note.sonots.com/SciSoftware/NcutImageSegmentation.html#sd872c22

Normalized Cut
http://note.sonots.com/SciSoftware/NcutImageSegmentation.html

segmentation (continued) - umartalhaumartalha.weebly.com/.../8382263/05_lecture_cv_nust_pnec.pdf ·...

Documents