06 - lff_iccv2009 recognizing multiple objects in an image - sharing and context

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 1/59

Multiclass object

detection

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 2/59

Multiclass object detection

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 3/59

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 4/59

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 5/59

Context: objects appear in configurations

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 6/59

Generalization: objects share parts

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 7/59

How many categories?

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 8/59

Slide by Aude Oliva

Muchas

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 9/59

How many object categories are there?

Biederman 1987

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 10/59

How many categories?

Probably this question is not even specificenough to have an answer

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 11/59

Which level of categorization

is the right one?

Car is an object composed of:

a few doors, four wheels (not all visible at all times), a roof,

front lights, windshield

If you are thinking in buying a car, you might want to be a bit more specific about

your categorization level.

?

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 12/59

Entry-level categories(Jolicoeur, Gluck, Kosslyn 1984)

Typical member of a basic-level category arecategorized at the expected level

Atypical members tend to be classified at a

subordinate level.

A birdAn ostrich

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 13/59

We do not need to recognize the exact category

A new class can borrow information from similar

categories

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 14/59

So, where is computer vision?

Well

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 15/59

Multiclass object detectionthe not so early days

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 16/59

Multiclass object detectionthe not so early days

Schneiderman-Kanade multiclass object detection

Using a set of independent binary classifiers was a common strategy:

Viola-Jones extension for dealing with rotations

- two cascades for each view

(a) One detector for each class

There is nothing wrong with this approach if you have access to

lots of training data and you do not care about efficiency.

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 17/59

Generalizing Across Categories

Can we transfer knowledge from one object category to another? Slide by Erik Sudderth

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 18/59

Shared features

Is learning the object class 1000 easier thanlearning the first?

Can we transfer knowledge from one object to

another?

Are the shared properties interesting bythemselves?

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 19/59

Multitask learningR. Caruana. Multitask Learning. ML 1997

MTL improves generalization by leveraging the domain-specific information contained

in the training signals of related tasks. It does this by training tasks in parallel while using

a shared representation.

vs.

Sejnowski & Rosenberg 1986; Hinton 1986; Le Cun et al. 1989; Suddarth & Kergosien

1990; Pratt et al. 1991; Sharkey & Sharkey 1992;

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 20/59

Multitask learning

horizontal location of doorknob

single or double door

horizontal location of doorway centerwidth of doorway

horizontal location of left door jamb

horizontal location of right door jamb

width of left door jamb

width of right door jambhorizontal location of left edge of door

horizontal location of right edge of door

Primary task: detect door knobs

Tasks used:

R. Caruana. Multitask Learning. ML 1997

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 21/59

Sharing invariancesS. Thrun.Is Learning the n-th Thing Any Easier Than Learning The First? NIPS 1996

Knowledge is transferred between tasks via a learned model of the invariances

of the domain: object recognition is invariant to rotation, translation, scaling,

lighting, These invariances are common to all object recognition tasks.

Toy world

Without sharing

With sharing

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 22/59

Convolutional Neural Network

Translation invariance is already built into the network

The output neurons share all the intermediate levels

Le Cun et al, 98

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 23/59

Sharing transformations

Miller, E., Matsakis, N., and Viola, P. (2000). Learning from one example through

shared densities on transforms. In IEEE Computer Vision and Pattern Recognition.

Transformations are shared

and can be learnt from other tasks.

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 24/59

Sharing in constellation models

Pictorial StructuresFi schler & Elschlager, IEEE Trans. Comp. 1973

Constellation ModelF ei-F ei  , F ergus, Perona, ICCV 2003

SVM DetectorsHei sele, Poggi o, et. al., NIPS 2001

Model-Guided SegmentationMor i  , Ren, Efros, & Mal i k, CVPR 2004

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 25/59

Reusable Parts

Goal: Look for a vocabulary of edges that reduces the number of 

features.

Krempp, Geman, & Amit Sequential Learning of Reusable Parts for Object Detection.

TR 2002

   N   u   m    b   e   r   o    f    f   e   a   t   u

   r   e   s

Number of classes

Examples of reused parts

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 26/59

Specific feature

 Non-shared feature: this feature

is too specific to faces.

 pedestrian

chair 

Traffic light

sign

face

Background class

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 27/59

Shared feature

shared feature

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 28/59

Additive models and boosting

Torralba, Murphy, Freeman. CVPR 2004. PAMI 2007

Screen detector 

Car detector 

Face detector 

Binary classifiers that share features:

Screen detector 

Car detector 

Face detector 

Independent binary classifiers:

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 29/59

50 training samples/class

29 object classes2000 entries in the dictionary

Results averaged on 20 runs

Error bars = 80% interval

Torralba, Murphy, Freeman. CVPR 2004. PAMI 2007

Shared features

Class-specific features

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 30/59

Generalization as a function of object

similarities

12 viewpoints12 unrelated object classes

Number of training samples per class Number of training samples per class

   A   r   e   a   u   n   d   e   r   R   O       C

   A   r   e   a   u   n   d   e   r   R   O       C

K = 2.1 K = 4.8

Torralba, Murphy, Freeman. CVPR 2004. PAMI 2007

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 31/59

Opelt, Pinz, Zisserman, CVPR 2006

Efficiency Generalization

J. Shotton, A. Blake, R. Cipolla.

Multi-Scale Categorical Object Recognition Using

Contour Fragments. In IEEETrans. on PAMI,

30(7):1270-1281, July 2008.

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 32/59

Sharing patches

Bart and Ullman, 2004For a new class, use only features similar to features that where good for other

classes:

Proposed Dog

features

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 33/59

Some more references

Baxter 1996

Caruana 1997

Schapire, Singer, 2000

Thrun, Pratt 1997

Krempp, Geman, Amit, 2002

E.L.Miller, Matsakis, Viola, 2000

Mahamud, Hebert, Lafferty, 2001

Fink et al. 2003, 2004 LeCun, Huang, Bottou, 2004

Holub, Welling, Perona, 2005

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 34/59

Modeling object

relationships

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 35/59

The guess what I am trying to detect challenge

The detector challenge: by looking at the output of a detector on a random set

of images, can you guess which object is it trying to detect?

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 36/59

What object is detector trying to

detect?

The detector challenge: by looking at the output of a detector on a random set

of images, can you guess which object is it trying to detect?

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 37/59

1. chair, 2. table, 3. road, 4. road, 5. table, 6. car, 7. keyboard.

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 38/59

The context challenge

How far can you go without

using an object detector?

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 39/59

21

What are the hidden objects?

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 40/59

What are the hidden objects?

Chance ~ 1/30000

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 41/59

p(O | I) Ep(I|O) p(O)

Object model Context model

imageobjects

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 42/59

p(O | I) Ep(I|O) p(O)

Object model Context model

Full jointScene model Aprox. joint

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 43/59

p(O | I) Ep(I|O) p(O)

Object model Context model

Full jointScene model Approx. joint

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 44/59

p(O | I) Ep(I|O) p(O)

Object model Context model

Full jointScene model

p(O) = 74p(Oi|S=s) p(S=s)s i

Approx. joint

office street

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 45/59

p(O | I) Ep(I|O) p(O)

Object model Context model

Full jointScene model Approx. joint

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 46/59

Pixel labeling using MRFs

Enforce consistency between neighboring labels,

and between labels and pixels

Carbonetto, de Freitas & Barnard, ECCV¶04

Oi

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 47/59

Beyond nearest-neighbor grids

Most MRF/CRF models assume nearest-neighbor graph topology

This cannot capture long-distance

correlations

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 48/59

Object-Object Relationships

Use latent variables to induce long distance correlationsbetween labels in a Conditional Random Field (CRF)

He, Zemel & Carreira-Perpinan (04)

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 49/59

Object-Object Relationships

[KumarHebert 2005]

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 50/59

Fink & Perona (NIPS 03)Use output of boosting from other objects at previous

iterations as input into boosting for this iteration

Object-Object Relationships

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 51/59

Objects in Context

Building,

boat, motorbike

Building, boat, person

Water,

sky

Road

Most consistent labeling

according to object co-

occurrences& locallabel

probabilities.

Boat

Building

Water

Road

A. Rabinovich, A. Vedaldi, C. Galleguillos, E. Wiewiora

and S. Belongie. Objects in Context. ICCV 2007

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 52/59

52

Objects in Context:

Contextual Refinement

Contextual model based on co-occurrences

Try to find the most consistent labeling with

high posterior probability and high mean

pairwise interaction.

Use CRF for this purpose.Boat

Building

Water

Road

Independent

segment classificationMean interaction of all label pairs

(i,j) is basically the observed label co-

occurrences in training set.

Slide by GokberkCinbis

d ff l b

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 53/59

Detecting difficult objects

Office Maybethere is

a mouse

Start recognizing the scene

Torralba, Murphy, Freeman. NIPS 2004.

i diffi l bj

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 54/59

Detecting difficult objects

Detect first simple objects (reliable detectors) that provide strong

contextual constraints to the target (screen -> keyboard -> mouse)

Torralba, Murphy, Freeman. NIPS 2004.

D i diffi l bj

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 55/59

Detecting difficult objects

Detect first simple objects (reliable detectors) that provide strong

contextual constraints to the target (screen -> keyboard -> mouse)

Torralba, Murphy, Freeman. NIPS 2004.

BRF f d i l

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 56/59

BRF for car detection: topology

Torralba Murphy Freeman (2004)

BRF f d i l

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 57/59

BRF for car detection: results

Torralba Murphy Freeman (2004)

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 58/59

A car out of context is less of a car

Car  Building Road

 b F G  b F G  b F G

From image

From detectors

Thresholded beliefs

C t t l bj t l ti hi

8/3/2019 06 - LFF_ICCV2009 Recognizing Multiple Objects in an Image - Sharing and Context

http://slidepdf.com/reader/full/06-lfficcv2009-recognizing-multiple-objects-in-an-image-sharing-and-context 59/59

Contextual object relationshipsCarbonetto, de Freitas & Barnard (2004) Kumar,Hebert (2005)

Torralba Murphy Freeman (2004)

Fink & Perona (2003)E. Sudderth et al (2005)