Slide 1

Perceptual Organization: Segmentation and Optical Flow

Slide 2

Inspiration from psychology The Gestalt school: Grouping is key to visual perception The whole is greater than the sum of its parts http://en.wikipedia.org/wiki/Gestalt_psychology subjective contours occlusion familiar configuration

Slide 3

Gestalt grouping factors

Slide 4

Emergence http://en.wikipedia.org/wiki/Gestalt_psychology

Slide 5

Motion and perceptual organization Even impoverished motion data can evoke a strong percept G. Johansson, Visual Perception of Biological Motion and a Model For Its Analysis", Perception and Psychophysics 14, 201-211, 1973. YouTube video

Slide 6

Image segmentation

Slide 7

The goals of segmentation Obtain primitives for other tasks Perceptual organization, recognition Graphics, image manipulation

Slide 8

Goal 1: Primitives for other tasks Group together similar-looking pixels for efficiency of further processing Bottom-up process Unsupervised X. Ren and J. Malik. Learning a classification model for segmentation. ICCV 2003.Learning a classification model for segmentation. superpixels

Slide 9

Image parsing or semantic segmentation: Segments as primitives for recognition J. Tighe and S. Lazebnik, ECCV 2010, IJCV 2013

Slide 10

Goal 2: Recognition Separate image into coherent objects Bottom-up or top-down process? Supervised or unsupervised? Berkeley segmentation database: http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/ http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/ image human segmentation

Slide 11

Goal 3: Image manipulation Interactive segmentation for graphics

Slide 12

Approaches to segmentation Segmentation as clustering Segmentation as graph partitioning Segmentation as labeling

Slide 13

Segmentation as clustering Source: K. Grauman

Slide 14

Image Intensity-based clustersColor-based clusters Segmentation as clustering K-means clustering based on intensity or color is essentially vector quantization of the image attributes Clusters dont have to be spatially coherent

Slide 15

Segmentation as clustering Source: K. Grauman

Slide 16

Segmentation as clustering Clustering based on (r,g,b,x,y) values enforces more spatial coherence

Slide 17

Segmentation as graph partitioning Node for every pixel Edge between every pair of pixels (or every pair of sufficiently close pixels) Each edge is weighted by the affinity or similarity of the two nodes w ij i j Source: S. Seitz

Slide 18

Measuring affinity Represent each pixel by a feature vector x and define an appropriate distance function small large Role of

Slide 19

Segmentation as graph partitioning Break Graph into Segments Delete links that cross between segments Easiest to break links that have low affinity similar pixels should be in the same segments dissimilar pixels should be in different segments ABC Source: S. Seitz w ij i j

Slide 20

Graph cut Set of edges whose removal makes a graph disconnected Cost of a cut: sum of weights of cut edges A graph cut gives us a segmentation What is a good graph cut and how do we find one? A B Source: S. Seitz

Slide 21

Minimum cut We can do segmentation by finding the minimum cut in a graph Efficient algorithms exist for doing this Minimum cut example

Slide 22

Minimum cut We can do segmentation by finding the minimum cut in a graph Efficient algorithms exist for doing this Minimum cut example

Slide 23

Normalized cut Drawback: minimum cut tends to cut off very small, isolated components Ideal Cut Cuts with lesser weight than the ideal cut * Slide from Khurram Hassan-Shafique CAP5415 Computer Vision 2003

Slide 24

Normalized cut To encourage larger segments, normalize the cut by the total weight of edges incident to the segment The normalized cut cost is: Intuition: big segments will have a large w(A,V), thus decreasing ncut(A, B) Finding the globally optimal cut is NP-complete, but a relaxed version can be solved using a generalized eigenvalue problem w(A, B) = sum of weights of all edges between A and B J. Shi and J. Malik. Normalized cuts and image segmentation. PAMI 2000Normalized cuts and image segmentation.

Slide 25

Normalized cut: Algorithm Let W be the affinity matrix of the graph (n x n for n pixels) Let D be the diagonal matrix with entries D(i, i) = j W(i, j) Solve generalized eigenvalue problem (D W)y = Dy for the eigenvector with the second smallest eigenvalue The ith entry of y can be viewed as a soft indicator of the component membership of the ith pixel Use 0 or median value of the entries of y to split the graph into two components To find more than two components: Recursively bipartition the graph Run k-means clustering on values of several eigenvectors

Slide 26

Example result

Slide 27

Challenge How to define affinities for segmenting highly textured images?

Slide 28

Segmenting textured images Convolve image with a bank of filters Find textons by clustering vectors of filter bank outputs J. Malik, S. Belongie, T. Leung and J. Shi. "Contour and Texture Analysis for Image Segmentation". IJCV 43(1),7-27,2001."Contour and Texture Analysis for Image Segmentation" Texton mapImage Filter bank

Slide 29

Segmenting textured images Convolve image with a bank of filters Find textons by clustering vectors of filter bank outputs Represent pixels by texton histograms computed over neighborhoods at some local scale Define affinities as similarities between local texton histograms J. Malik, S. Belongie, T. Leung and J. Shi. "Contour and Texture Analysis for Image Segmentation". IJCV 43(1),7-27,2001."Contour and Texture Analysis for Image Segmentation"

Slide 30

Pitfall of texture features Possible solution: check for intervening contours when computing affinities J. Malik, S. Belongie, T. Leung and J. Shi. "Contour and Texture Analysis for Image Segmentation". IJCV 43(1),7-27,2001."Contour and Texture Analysis for Image Segmentation"

Slide 31

Results: Berkeley Segmentation Engine http://www.cs.berkeley.edu/~fowlkes/BSE/

Slide 32

Berkeley Segmentation Engine http://www.cs.berkeley.edu/~fowlkes/BSE/

Slide 33

Pro Generic framework, can be used with many different features and affinity formulations Con High storage requirement and time complexity: involves solving a generalized eigenvalue problem of size n x n, where n is the number of pixels Normalized cuts: Pro and con

Slide 34

Efficient graph-based segmentation Runs in time nearly linear in the number of edges Easy to control coarseness of segmentations Results can be unstable P. Felzenszwalb and D. Huttenlocher, Efficient Graph-Based Image Segmentation, IJCV 2004Efficient Graph-Based Image Segmentation

Slide 35

Segmentation as labeling Suppose we want to segment an image into foreground and background Binary labeling problem Credit: N. Snavely

Slide 36

Segmentation as labeling Suppose we want to segment an image into foreground and background Binary labeling problem User sketches out a few strokes on foreground and background How do we label the rest of the pixels? Source: N. Snavely

Slide 37

Binary segmentation as energy minimization Define a labeling L as an assignment of each pixel with a 0-1 label (background or foreground) Find the labeling L that minimizes data term smoothness term How similar is each labeled pixel to the foreground or background? Encourage spatially coherent segments Source: N. Snavely

Slide 38

: distance from pixel to background : distance from pixel to foreground { computed by creating a color model from user- labeled pixels Source: N. Snavely

Slide 39

Slide 40

Neighboring pixels should generally have the same labels Unless the pixels have very different intensities : similarity in intensity of p and q = 10.0 = 0.1 Source: N. Snavely

Slide 41

Binary segmentation as energy minimization For this problem, we can efficiently find the global minimum using the max flow / min cut algorithm Source: N. Snavely Y. Boykov and M.-P. Jolly, Interactive Graph Cuts for Optimal Boundary and Region Segmentation of Objects in N-D Images, ICCV 2001Interactive Graph Cuts for Optimal Boundary and Region Segmentation of Objects in N-D Images

Slide 42

Recall: Stereo as energy minimization I1I1 I2I2 D Energy functions of this form can be minimized using graph cuts Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 2001Fast Approximate Energy Minimization via Graph Cuts W1(i )W1(i )W 2 (i+D(i )) D(i )D(i ) data term smoothness term

Slide 43

GrabCut C. Rother, V. Kolmogorov, and A. Blake, GrabCut Interactive Foreground Extraction using Iterated Graph Cuts, SIGGRAPH 2004GrabCut Interactive Foreground Extraction using Iterated Graph Cuts