co-segmentation

68
Co-Segmentation Presented By : Murad Tukan

Upload: odina

Post on 01-Feb-2016

91 views

Category:

Documents


0 download

DESCRIPTION

Co-Segmentation. Presented By : Murad Tukan. Introduction. Today there is a massive attempt to exclude the same object from different images. Such problem is not an easy task as it seems , furthermore the algorithm which is presented today is not 100% accurate even though it is efficient. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Co-Segmentation

Co-Segmentation Presented By : Murad Tukan

Page 2: Co-Segmentation

Introduction

• Today there is a massive attempt to exclude the same object from different images.

• Such problem is not an easy task as it seems , furthermore the algorithm which is presented today is not 100% accurate even though it is efficient.

Page 3: Co-Segmentation

What Are We Going To Learn ?

• An Efficient Algorithm for Co-segmentation .

• Clothing Co-segmentation for Recognizing People .

• iCoseg: Interactive Co-segmentation on your iOS device (in short).

• Implementation of Co-segmentation in MATLAB + C++ Code.

Page 4: Co-Segmentation

An Efficient Algorithm for Co-segmentation

Page 5: Co-Segmentation

• The Goal of the Algorithm: Given a set of images , we need to detect the same

foreground in the set of images despite the differences in the background.

• Approach : We will reward consistency in the two foreground histograms.

Motivation

Page 6: Co-Segmentation

1

2

Images Histogram (Intensity)

Page 7: Co-Segmentation

Definitions

• Co-segmentation settings.

• Similarity weight.

• Foreground and background pixels.

• “MRF penalties”.

Page 8: Co-Segmentation

Co-segmentation settings

Page 9: Co-Segmentation

Co-segmentation settings (Cont.)

Page 10: Co-Segmentation

Similarity Weight

Page 11: Co-Segmentation

Example“image 1”: “image 2”:

B1 : B2 :

Page 12: Co-Segmentation

Foreground and background pixels

• The segmentation of each image will partition the set of pixel into foreground versus background pixels.

• Our goal is to ensure that the foreground in the two images are similar.

Page 13: Co-Segmentation

Foreground and background pixels(cont.)

Page 14: Co-Segmentation

Foreground and background pixels(cont.)

Page 15: Co-Segmentation

“MRF penalties”

Page 16: Co-Segmentation

“MRF penalties” (cont.)

• The MRF formulation of one image is then :

min 𝑑𝑗 ∗𝑥𝑗 + 𝑤𝑖𝑗 ∗𝑦𝑖𝑗𝑖~𝑗 𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑥𝑖 − 𝑥𝑗 ≤ 𝑦𝑖𝑗 𝑥𝑗 − 𝑥𝑖 ≤ 𝑦𝑗𝑖 𝑥𝑖,𝑦𝑖𝑗 𝑏𝑖𝑛𝑎𝑟𝑦 𝑓𝑜𝑟 𝑖,𝑗= 1,2,…,𝑛 𝑤ℎ𝑒𝑟𝑒 𝑑𝑗 = 𝑔𝑗 − 𝐼𝑗

Deviation penalty Seperation penalty

Page 17: Co-Segmentation

Problem Statement

Page 18: Co-Segmentation

Problem Statement (cont.)

Page 19: Co-Segmentation

Problem Statement (cont.)

Page 20: Co-Segmentation

Problem Statement (cont.)

• Therefore the formulation of co-segmentation problem is then a linear combination of MRF minimization and similarity maximization:

min 𝑑𝑗ሺ1ሻ∗𝑥𝑗ሺ1ሻ+ 𝑤𝑖𝑗 ∗𝑦𝑖𝑗ሺ1ሻ+ 𝑑𝑗ሺ2ሻ∗𝑥𝑗ሺ2ሻ+ 𝑤𝑖𝑗 ∗𝑦𝑖𝑗ሺ2ሻ− 𝜆∗ 𝑧𝑖𝑗𝑖∈𝐼ሺ1ሻ∩𝐻𝑘 , 𝑗∈𝐼ሺ2ሻ∩𝐻𝑘𝐾

𝑘=1 𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑧𝑖𝑗 ≤ 𝑥𝑖ሺ1ሻ 𝑓𝑜𝑟 𝑖 ∈𝐼ሺ1ሻ

𝑧𝑖𝑗 ≤ 𝑥𝑗ሺ2ሻ 𝑓𝑜𝑟 𝑗∈𝐼ሺ2ሻ 𝑥𝑖ሺ𝑞ሻ− 𝑥𝑗ሺ𝑞ሻ≤ 𝑦𝑖𝑗 𝑓𝑜𝑟 𝑞 = 1,2 𝑥𝑗ሺ𝑞ሻ− 𝑥𝑖ሺ𝑞ሻ≤ 𝑦𝑗𝑖 𝑓𝑜𝑟 𝑞 = 1,2 𝑥𝑖ሺ𝑞ሻ,𝑦𝑖,𝑗ሺ𝑞ሻ,𝑧𝑖𝑗 𝑏𝑖𝑛𝑎𝑟𝑦 𝑓𝑜𝑟 𝑞 = 1,2 𝑖,𝑗= 1,…,𝑛

Page 21: Co-Segmentation

The graph construction

Page 22: Co-Segmentation

The graph construction (cont.)

Page 23: Co-Segmentation

The graph construction (cont.)

Page 24: Co-Segmentation
Page 25: Co-Segmentation

The Algorithm (Co-Seg)

Page 26: Co-Segmentation
Page 27: Co-Segmentation

Histogram Buckets

Page 28: Co-Segmentation

Summarizing

•We have seen an efficient algorithm, running in polynomial time (the running time of a max-flow algorithm).

Page 29: Co-Segmentation

Advantages through Comparison

Page 30: Co-Segmentation

Clothing Co-segmentation for recognizing people

Page 31: Co-Segmentation

Motivation

• The goal: Given a four consumer image collection , we need to recognize

individuals among these image collection.

• Approach: Using the clothing data.

Page 32: Co-Segmentation

How many different faces do you recognize?

Page 33: Co-Segmentation

Reminders

• Graph Cut.

• Normalized Cut.

• Texture feature vector.

Page 34: Co-Segmentation

Graph Cut

• What are Graph-Cuts ? Set of edges whose removal makes a graph disconnected.

It’s the partition of V into A,B such that and

• A graph can be partitioned into two disjoint sets ,we define the partition cost as:

• The bipartition of the graph which minimizes the cut value is called the Min-Cut .

VBA ,BA

),( EVG

BvAu

vuwBACut,

).,(),(

Page 35: Co-Segmentation

Normalized cut (Ncut)

• Cut(A,B) is sum of weights with one end in A and one end in B ,we want to minimize the cut cost.

• Assoc(A,V) is sum of all edges with one end in A , we want to maximize the sum of all weights for every A,B element in the partition

),(

),(

),(

),(),(

VBassoc

BAcut

VAassoc

BAcutBANcut

BvAu

vuwBACut,

).,(),(

.),(),(,

VtAu

tuwVAassoc

Page 36: Co-Segmentation

texture feature vector

• Feature vector based on texture segmentation:

• We can use spatial filter for where the DOOG filters at various scales and orientations .

]*,,*[)( 1 nfIfIiF

Page 37: Co-Segmentation

Luminance-chrominance space (LCC)

• The color encoding system used for analog television worldwide (NTSC, PAL and SECAM). The YUV color space (color model) differs from RGB, which is what the camera captures and what humans view. 

• The Y in YUV stands for "luma," which is brightness.

• U and V provide color information and are "color difference" signals of blue minus luma (B-Y) and red minus luma (R-Y)

Page 38: Co-Segmentation

Luminance-chrominance space (LCC) (cont.)

• RGB to LCC / YUV :

Page 39: Co-Segmentation

Images and Features for Clothing Analysis

• Features are extracted from the faces and clothing of people:

• Face features:

• Are extracted using an algorithm of detecting face which also can estimate the position of the eyes.

• Each face is normalized in scale (49*61 pixels) and projected onto a set of Fisherfaces , representing each face as a 37-dimentional vector.

Page 40: Co-Segmentation

Fisherfaces• A N x N pixel image of a face, represented

as a vector occupies a single point in N2-dimensional image space.

• Images of faces being similar in overall configuration, will not be randomly distributed in this huge image space.

• Therefore, they can be described by a low dimensional subspace.

• Main idea of PCA for faces:– To find vectors that best account for

variation of face images in entire image space.

– These vectors are called eigen vectors.– Construct a face space and project the

images into this face space (eigenfaces).

Page 41: Co-Segmentation

Images and Features for Clothing Analysis (cont.)

Page 42: Co-Segmentation

90, 133, 176,0,0

Page 43: Co-Segmentation

Images and Features for Clothing Analysis (cont.)

Page 44: Co-Segmentation
Page 45: Co-Segmentation
Page 46: Co-Segmentation

Images and Features for Clothing Analysis (cont.)

2:

oThe 5-dimentional vector at each pixel is quantized to the index of the closest visual word , so each pixel in the region is associated with a word in the dictionary (or several words) then all the words of all the pixels in the region are counted in a histogram.

oThe Histogram per region is the "feature" of the region. The visual word clothing features are represented as v.

Page 47: Co-Segmentation

Example of visual words:- red- dots- green …

red,dot

Page 48: Co-Segmentation

Finding the Global Clothing Mask

Page 49: Co-Segmentation

Finding the Global Clothing Mask (cont.)

Page 50: Co-Segmentation

Finding the Global Clothing Mask (cont.)

• The same day mutual information maps are reflected (symmetry is assumed) , summed and thresholded (by a value constant across image collection) to yield clothing masks that appear remarkably similar across collections.

Page 51: Co-Segmentation

Graph Cuts for Clothing Segmentation

Page 52: Co-Segmentation

Graph Cuts for Clothing Segmentation( cont.)

Page 53: Co-Segmentation

Graph Cuts for Clothing Segmentation( cont.)

Page 54: Co-Segmentation
Page 55: Co-Segmentation
Page 56: Co-Segmentation

Recognizing people

Page 57: Co-Segmentation

Recognizing people (cont.)

Page 58: Co-Segmentation
Page 59: Co-Segmentation

Experiments

• Better Clothing Recognition Improves Recognition:

• We use nine nearest neighbors and repeat random labeling procedure 50 times to the average performance.

Page 60: Co-Segmentation

Summary

• The identity recognition and clothing segmentation problems are inter-twined; a good solution for one aides in the solution for the other.

• Multiple images of the same person improves clothing segmentation.

• Person recognition improves with improvements to the clothing segmentation.

Page 61: Co-Segmentation

iCoseg: Interactive Co-segmentation on your iOS device

Page 62: Co-Segmentation

Interactive Co-segmentation

Page 63: Co-Segmentation

Embedded Devices

Page 64: Co-Segmentation

iCoseg for iPhone, iPod Touch and iPad

Page 65: Co-Segmentation
Page 66: Co-Segmentation

References

• Dorit S. Hochbaum, Vikas Singh. An efficient algorithm for co-segmentation. ICCV Oct 2009 , Kyoto, Japan, the 12th IEEE Conference on Computer Vision. 

• A. C. Gallagher and T. Chen. Clothing cosegmentation for recognizing people. In Proc. of Conf. on Computer Vision and Pattern Recognition, 2008.

• Slides Credit: Jad Silbak

• http://en.wikipedia.org/wiki/YUV

• http://chenlab.ece.cornell.edu/people/jasonlew/Poster_JasonLew.pdf

Page 67: Co-Segmentation
Page 68: Co-Segmentation