Category Independent Region Proposals

Ian Endres and Derek HoiemUniversity of Illinois at Urbana-Champaign

Finding Objects

Scanning Window

HorseDogCatCarTrain… 10,000+ windows

Category Independent Search

~100 regions

Finding Unfamiliar Objects

Finding Objects

Objectives:1. Minimize number of proposed regions2. Maintain high recall of all objects3. Provide detailed spatial support (i.e. segmentation)

Challenges

• Objects extremely diverse– Variety of shapes, sizes– Many different appearances

• Within object variation– Multiple materials and textures– Strong interior boundaries

• Many objects in an image

Overview

1 2 3 4 ...

Generate Proposals:Maximize recall

Rank Proposals:Small diverse set of object regions

Generating Proposals1. Select Seed 2. Compute affinities for seed

3. Construct binary CRF

+

Unary term:Affinities

Pairwise term:Occlusion Boundaries

4. Compute proposal

5. Change parametersRepeat

Generating Seeds

• Compute occlusion boundaries (Hoiem et al. ICCV ‘07)

• Generate hierarchal segmentation– Incrementally merge regions of oversegmentation

• Use regions with sufficient size and boundary strength– Avoids redundant or uninformative seeds

Region Affinity

• Learned from pairs of regions belonging to an object– Computed between the seed and each region of

the hierarchy

– Features: color and texture similarity, boundary crossings, layout agreement

Color/Texture Similarity•Color, texture histograms for each region•Compute histogram intersection distance between two regions

Boundary Crossing•Draw line between region centers of mass

•Compute strength of occlusion boundaries crossed

Layout Agreement•Predict object extent from each region

•Compute strength of agreement between two regions

CRF Segmentation

• Binary segmentation• Graph composition:– Nodes: Superpixels– Edges: Adjacent superpixels

+

CRF Segmentation

• Graph Potentials– Unary Potential: affinity values for each superpixel– Edge Potential: occlusion boundary strength

• Parameters (25 combinations)– Node/Edge weight tradeoff– Node bias

+

Unary potential:Affinities

Edge potential:Occlusion Boundaries

Ranking Proposals

wT X1

wT X3

Appearance scores

wT X4

1.

2.

3.

4.

wT X2Sort

scores

GeneratedRanking

Lacks Diversity

• But in an image with many objects, one object may dominate 1

2

3

4

…

20

150

100

…

50…

…

Encouraging Diversity

• Suppress regions with high overlap with previous proposals

…

1

2

3

10

4

…

20

50

100

…

…

Ranking as Structured Prediction

• Find the max scoring ordering of proposals

• Greedily add proposals with best overall score

Appearance score

Overlap penalty

Gives higher weight to higher ranked proposals

Overall score

Learning to Rank(Max-margin Structured Learning)

• Score of ground truth ordering (R(n)) should be greater than all other orderings (R):

• Loss ( ) encourages good orderings:– Higher quality proposals should have higher rank– Each object should have a highly ranked proposal

Experimental Setup• Train on 200 BSDS images

• Test 1: 100 BSDS images

• Test 2: 512 Images from Pascal 2008 Seg. Val.

Evaluation

• Region overlap

• Recall at 50% region overlap– Typically more strict that 50% bounding box overlap– Measures detection quality and segment quality

Ai Aj

Qualitative Results

Pascal

BSDS(Rank, % overlap)

Vs. Standard Segmentation

Standard: 53%3000 proposals

Ours: 53%18 proposals

Standard: 80%70,000 proposals

(merge 2 adjacent regions)

Ours: 80%180 proposals

Recalling Pascal Categories

Future work

• Object Discovery• Incorporate into detection systems– Label regions directly– Voting from proposed regions

• Refine proposals with domain knowledge– i.e. wheel or head models