Andrew ZissermanVisual Geometry Group

University of Oxfordhttp://www.robots.ox.ac.uk/~vgg

Includes slides from: Mark Everingham, Pedro Felzenszwalb, Rob Fergus, Kristen Grauman, Bastian Leibe, Fei-Fei Li, Marcin Marszalek, Pietro Perona, Deva Ramanan, Josef Sivic and Andrea Vedaldi

Visual search and recognitionPart II – category recognition

What we would like to be able to do …

• Visual recognition and scene understanding• What is in the image and where

• scene type: outdoor, city …• object classes• material properties• actions

Recognition Tasks• Image Classification

– Does the image contain an aeroplane?

• Object Class Detection/Localization– Where are the aeroplanes (if any)?

• Object Class Segmentation– Which pixels are part of an aeroplane

(if any)?

Things vs. StuffStuff (n): Material defined by a homogeneous or repetitive pattern of fine-scale properties, but has no specific or distinctive spatial extent or shape.

Thing (n): An object with a specific size and shape.

Ted Adelson, Forsyth et al. 1996.

Slide: Geremy Heitz

Challenges: Clutter

Challenges: Occlusion and truncation

Challenges: Intra-class variation

Object Category Recognition by Learning• Difficult to define model of a category. Instead, learn from

example images

Level of Supervision for LearningImage-level label

Pixel-level segmentation

Bounding box

“Parts”

Outline1. Image Classification

• Bag of visual words method

• Features and adding spatial information

• Encoding

• PASCAL VOC and other datasets

2. Object Category Detection

3. The future and challenges

Recognition Task• Image Classification

– Does the image contain an aeroplane?

• Challenges– Imaging factors e.g. lighting, pose,

occlusion, clutter– Intra-class variation

– Position can vary within image– Training data may not specify

Image classification

• Supervised approach – Training data with labels indicating presence/absence of the class

Positive training images containing an object class, here motorbike

Negative training images that don’t

– Learn classifier

Image classification

• Test image– Image without label– Determine whether test image contains the object class or not

• Classify– Run classifier on the test image

?

Bag of visual words• Images yield varying number of local

features• Features are high-dimensional

e.g. 128-D for SIFT

• How to summarize image content in a fixed-length vector for classification?

1. Map descriptors onto a common vocabulary of visual words

2. Represent image as a histogram over visual words – a bag of words

Examples for visual words

Airplanes

Motorbikes

Faces

Wild Cats

Leaves

People

Bikes

Intuition

Visual Vocabulary

• Visual words represent “iconic” image fragments• Discarding spatial information gives lots of invariance

positive negative

Train classifier,e.g. SVM

Training data: vectors are histograms, one from each training image

Faces 435

Motorbikes 800

Airplanes 800

Cars (rear) 1155

Background 900

Total: 4090

Example Image collection: four object classes + background

The “Caltech 5”

Example: weak supervision

Training• 50% images• No identifcation of

object within image

Testing• 50% images• Simple object

present/absent test

Motorbikes Airplanes Frontal Faces

Cars (Rear) Background

Learning• SVM classifier• Gaussian kernel using as similarity between histograms

Result• Between 98.3 – 100% correct, depending on class

Zhang et al 2005Csurka et al 2004

K(x,y) = e−γχ2(x,y)

Localization according to visual word probability

50 100 150 200

20

40

60

80

100

120

50 100 150 200

20

40

60

80

100

120

foreground word more probable

background word more probable

sparse segmentation

Why does SVM learning work?

• Learns foreground and background visual words

foreground words – positive weight

background words – negative weight

w

Linear SVM, f(x) = w>x+ b

Bag of visual words summary

• Advantages:– largely unaffected by position and orientation of object in image– fixed length vector irrespective of number of detections– Very successful in classifying images according to the objects they

contain

• Disadvantages:– No explicit use of configuration of visual word positions– Poor at localizing objects within an image

Adding Spatial Information

Beyond BOW II: Grids and spatial pyramidsStart from BoW for image

• no spatial information recorded

Bag of Words

Feature Vector

Adding Spatial Information to Bag of Words

Bag of Words

Concatenate

Feature Vector[Fergus et al, 2005]Keeps fixed length feature vector for a window

Tiling defines (records) the spatial correspondence of the words

If codebook has V visual words, then representation has dimension 4V

Fergus et al ICCV 05

• parameter: number of tiles

Spatial Pyramid – represent correspondence

•••••••••••••••••••••

1 BoW

4 BoW

16 BoW

[Lazebnik et al, 2006][Grauman & Darrell, 2005]

Dense Visual Words• Why extract only sparse visual words?• Good where lots of invariance is needed (e.g. to rotation or

scale), but not relevant if it isn’t• Also, interest points do not necessarily capture “all” features

• Instead, extract dense visual words of fixed scales on an overlapping grid

• More “detail” at the expense of invariance• Improves performance for most categories• Pyramid histogram of visual words (PHOW)

[Luong & Malik, 1999][Varma & Zisserman, 2003]

[Vogel & Schiele, 2004][Jurie & Triggs, 2005]

[Fei-Fei & Perona, 2005][Bosch et al, 2006]Patch / SIFT

QuantizeWord

Image categorization: summary

... ... ... ...

VQ

Linear SVM

dogs

Example – retrieving videos of scene categories

• TrecVid 2010 test data

– 8383 videos

– 144,988 shots

– 232,276 key-frames

– 200G mpeg2

Feature ExtractionDense Sift + VQ+ Spatial Tiling

Positive (Airplane)

Negative (Background)

Training Images Features

LearningSupport VectorMachine (SVM)

Test Images

Feature ExtractionDense Sift + VQ + Spatial Tiling

Features

Scoring

Scores

0.91

0.12

0.65

0.89

Ranked List

0.91

0.89

0.65

0.12

Ranking

Classifier Model

Test

ing

Trai

ning

Example – retrieving videos of scene categories

aircraft

Example – retrieving videos of scene categories

cityscape

Example – retrieving videos of scene categories

demonstration

Example – retrieving videos of scene categories

singing

Encodings

Beyond hard assignments …1. Distance based methods

– Represent descriptor by function of distance to set of nearest neighbour cluster centres (soft assignment)

A: 0.1B: 0.5C: 0.4

B: 1.0 Hard Assignment

Soft Assignment

[Philbin et al. CVPR 2008, Van Gemert et al. ECCV 2008]

Beyond hard assignments …2. Reconstruction based methods

– Sparse-coding: approximate descriptor using nearest neighbourcentres as a basis

– Locality-constrained Linear Coding (LLC)

[Wang et al. CVPR 2010]

minα||x− Bα||2 + λ||α||2

where B is a matrix formed from the nearest centres as column vectors

Beyond hard assignments …3. Represent the residual (and more)

– Measure (and quantize) the difference vector from the cluster centre

[Jegou et al. CVPR 2010, Perronin et al., Fisher kernels ECCV 2010 ]

ci

x

Spatial tiling & binning

1. Aggregate by sum-pooling

2. Aggregate by max-pooling

[Wang et al. CVPR 2010]

sum

max

Datasets

The PASCAL Visual Object Classes (VOC) Dataset and Challenge

Mark EveringhamLuc Van GoolChris Williams

John WinnAndrew Zisserman

• Challenge in visual objectrecognition funded byPASCAL network of excellence

• Publicly available dataset ofannotated images

• Main competitions in classification (is there an X in this image), detection (where are the X’s), and segmentation (which pixels belong to X)

• “Taster competitions” in 2-D human “pose estimation” (2007-present) and static action classes (2010-present)

• Standard evaluation protocol (software supplied)

The PASCAL VOC Challenge

Dataset Content

• 20 classes: aeroplane, bicycle, boat, bottle, bus, car, cat, chair, cow, dining table, dog, horse, motorbike, person, potted plant, sheep, train, TV

• Real images downloaded from flickr, not filtered for “quality”

• Complex scenes, scale, pose, lighting, occlusion, ...

Annotation• Complete annotation of all objects

• Annotated in one session with written guidelines

TruncatedObject extends beyond BB

OccludedObject is significantly occluded within BB

PoseFacing left

DifficultNot scored in evaluation

Examples

Aeroplane

Bus

Bicycle Bird Boat Bottle

Car Cat Chair Cow

Examples

Dining Table

Potted Plant

Dog Horse Motorbike Person

Sheep Sofa Train TV/Monitor

Dataset Collection• Images downloaded from flickr

– 500,000 images downloaded and random subset selected for annotation

– Queries• Keyword e.g. “car”, “vehicle”, “street”, “downtown”• Date of capture e.g. “taken 21-July”

– Removes “recency” bias in flickr results• Images selected from random page of results

– Reduces bias toward particular flickr users

• 2008/9 datasets retained as subset of 2010– Assignments to training/test sets maintained

Dataset Statistics• Around 40% increase in size over VOC2009

22,992

9,637

23,374

10,103

Training Testing

Images (7,054) (6,650)

Objects (17,218) (16,829)

VOC2009 counts shown in brackets

• Minimum ~500 training objects per category– ~1700 cars, 1500 dogs, 7000 people

• Approximately equal distribution across training and test sets

Classification Challenge• Predict whether at least one object of a given class is present in an image

• Evaluation: average precision per class

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

recall

prec

isio

n

AP

• Average Precision (AP) measures area under precision/recall curve

• Application independent

• A good score requires both high recall and high precision

Precision/Recall: Aeroplane (All)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

recall

prec

isio

nAll results

NEC_V1_HOGLBP_NONLIN_SVM (93.3)NEC_V1_HOGLBP_NONLIN_SVMDET (93.3)NUSPSL_KERNELREGFUSING (93.0)NUSPSL_MFDETSVM (91.9)CVC_PLUSDET (91.7)UVA_BW_NEWCOLOURSIFT (91.5)NUSPSL_EXCLASSIFIER (91.3)CVC_PLUS (91.0)SURREY_MK_KDA (90.6)UVA_BW_NEWCOLOURSIFT_SRKDA (90.6)NLPR_VSTAR_CLS_DICTLEARN (90.3)BUT_FU_SVM_SIFT (89.7)CVC_FLAT (89.4)BONN_FGT_SEGM (88.0)LIRIS_MKL_TRAINVAL (87.5)TIT_SIFT_GMM_MKL (87.2)XRCE_IFV (87.1)NUDT_SVM_LDP_SIFT_PMK_SPMK (86.1)RITSU_CBVR_WKF (85.6)UC3M_GENDISC (85.5)NUDT_SVM_WHGO_SIFT_CENTRIST_LLM (83.5)BUPT_LPBETA_MULTFEAT (82.1)BUPT_SVM_MULTFEAT (81.1)BUPT_SPM_SC_HOG (79.6)LIP6UPMC_RANKING (78.8)LIP6UPMC_MKL_L1 (78.5)LIP6UPMC_KSVM_BASELINE (78.4)NTHU_LINSPARSE_2 (77.9)WLU_SPM_EMDIST (75.8)LIG_MSVM_FUSE_CONCEPT (74.4)NII_SVMSIFT (69.3)HIT_PROTOLEARN_2 (60.7)

Precision/Recall: Aeroplane (Top 10 by AP)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

recall

prec

isio

nTop 10 results by AP

NEC_V1_HOGLBP_NONLIN_SVM (93.3)NEC_V1_HOGLBP_NONLIN_SVMDET (93.3)NUSPSL_KERNELREGFUSING (93.0)NUSPSL_MFDETSVM (91.9)CVC_PLUSDET (91.7)UVA_BW_NEWCOLOURSIFT (91.5)NUSPSL_EXCLASSIFIER (91.3)CVC_PLUS (91.0)SURREY_MK_KDA (90.6)UVA_BW_NEWCOLOURSIFT_SRKDA (90.6)

AP by Class

0102030405060708090

100ae

ropl

ane

pers

ontra

inbu

sm

otor

bike

hors

eca

rca

tbi

cycl

ebo

attv

/mon

itor

bird

dog

shee

pco

wch

air

dini

ngta

ble

sofa

bottl

epo

ttedp

lant

AP

(%) max

medianchance

• Max AP: 93.3% (aeroplane) ... 53.3% (potted plant)

Progress 2008-2010

• Results on 2008 data improve for best 2009 and 2010 methods for all categories, by over 100% for some categories

– Caveat: Better methods or more training data?

0102030405060708090

100ae

ropl

ane

bicy

cle

bird

boat

bottl

ebu

sca

r

cat

chai

rco

wdi

ning

tabl

edo

g

hors

em

otor

bike

pers

onpo

ttedp

lant

shee

p

sofa

train

tvm

onito

r

Max

AP

(%)

200820092010

The Indoor Scene Dataset

• 67 indoor categories

• 15620 images

• At least 100 images per category

• Training 67 x 80 images

• Testing 67 x 20 images

• A. Quattoni, and A.Torralba. Recognizing Indoor Scenes. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.

The Oxford Flowers Dataset• Explore fine grained visual categorization

• 102 different species

Dataset statistics• 102 categories

• Training set – 10 images per category

• Validation set– 10 images per category

• Test set– >20 images per category. – Total 6129 images.

Fine grained visual classification – flowers

Y. Chai, M.E. Nilsback, V. Lempitsky, A. Zisserman, ICVGIP’08, ICCV’ 11

Outline1. Image Classification

2. Object Category Detection

• Sliding window methods

• Histogram of Oriented Gradients (HOG)

• Learning an object detector

• PASCAL VOC (again) and two state of the art algorithms

3. The future and challenges

• Object Class Detection/Localization– Where are the aeroplanes (if any)?

Recognition Task

• Challenges– Imaging factors e.g. lighting, pose,

occlusion, clutter– Intra-class variation

• Compared to Classification– Detailed prediction e.g. bounding box– Location usually provided for training

aeroplane bicycle

car cow

motorbikehorse

Preview of typical results

Problem of background clutter• Use a sub-window

– At correct position, no clutter is present– Slide window to detect object– Change size of window to search over scale

Yes,a carNo,

not a car

Detection by Classification• Basic component of sliding window classifier: binary classifier

Car/non-carClassifier

Detection by Classification• Detect objects in clutter by search

Car/non-carClassifier

• Sliding window: exhaustive search over position and scale

Detection by Classification• Detect objects in clutter by search

Car/non-carClassifier

• Sliding window: exhaustive search over position and scale

Detection by Classification• Detect objects in clutter by search

Car/non-carClassifier

• Sliding window: exhaustive search over position and scale(can use same size window over a spatial pyramid of images)

Window (Image) Classification

• Features usually engineered• Classifier learnt from data

FeatureExtraction

Classifier

Training Data

Car/Non-car

Problems with sliding windows …

• aspect ratio

• granuality (finite grid)

• partial occlusion

• multiple responses

See work by

• Christoph Lampert et al CVPR 08, ECCV 08

Bag of (visual) Words representation

• Detect affine invariant local features (e.g. affine-Harris)

• Represent by high-dimensionaldescriptors, e.g. 128-D for SIFT

• Summarizes sliding window content in a fixed-length vector suitable for classification

1. Map descriptors onto a common vocabulary of visual words

2. Represent sliding window as a histogram over visual words – a bag of words

Sliding window detector• Classifier: SVM with linear kernel

• BOW representation for ROI

Example detections for dog

Lampert et al CVPR 08: Efficient branch and bound search over all windows

Discussion: ROI as a Bag of Visual Words

• Advantages– No explicit modelling of spatial information ⇒

high level of invariance to position and orientation in image

– Fixed length vector ⇒ standard machine learning methods applicable

• Disadvantages– No explicit modelling of spatial information ⇒

less discriminative power– Inferior to state of the art performance– Add dense features

Dalal & Triggs CVPR 2005Pedestrian detection

• Objective: detect (localize) standing humans in an image

• Sliding window classifier

• Train a binary classifier on whether a window contains a standing person or not

• Histogram of Oriented Gradients (HOG) feature

• Although HOG + SVM originally introduced for pedestrians has been used very successfully for many object categories

Feature: Histogram of Oriented Gradients (HOG)

imagedominant direction HOG

frequ

ency

orientation

• tile 64 x 128 pixel window into 8 x 8 pixel cells

• each cell represented by histogram over 8 orientation bins (i.e. angles in range 0-180 degrees)

Histogram of Oriented Gradients (HOG) continued

• Adds a second level of overlapping spatial bins re-normalizing orientation histograms over a larger spatial area

• Feature vector dimension (approx) = 16 x 8 (for tiling) x 8 (orientations) x 4 (for blocks) = 4096

Window (Image) Classification

• HOG Features• Linear SVM classifier

FeatureExtraction

Classifier

Training Data

pedestrian/Non-pedestrian

Averaged examples

Advantages of linear SVM:

• Training (Learning)• Very efficient packages for the linear case, e.g. LIBLINEAR for batch training and Pegasos for on-line training.

• Complexity O(N) for N training points (cf O(N^3) for general SVM)

• Testing (Detection)

Classifier: linear SVMf(x) = w>x+ b

f(x) =SXi

αik(xi,x) + b

f(x) =SXi

αixi>x+ b

= w>x+ b

S = # of support vectors

= (worst case ) N

size of training data

Non-linear

Linear

Independent of size of training data

Dalal and Triggs, CVPR 2005

Learned model

f(x) = w>x+ b

average over positive training data

Slide from Deva Ramanan

Why does HOG + SVM work so well?• Similar to SIFT, records spatial arrangement of histogram orientations• Compare to learning only edges:

– Complex junctions can be represented– Avoids problem of early thresholding– Represents also soft internal gradients

• Older methods based on edges have become largely obsolete

• HOG gives fixed length vector for window, suitable for feature vector for SVM

Training a sliding window detector• Object detection is inherently asymmetric: much more

“non-object” than “object” data

• Classifier needs to have very low false positive rate• Non-object category is very complex – need lots of data

Bootstrapping

1. Pick negative training set at random

2. Train classifier3. Run on training data4. Add false positives to

training set5. Repeat from 2

• Collect a finite but diverse set of non-object windows• Force classifier to concentrate on hard negative examples

• For some classifiers can ensure equivalence to training on entire data set

Example: train an upper body detector– Training data – used for training and validation sets

• 33 Hollywood2 training movies• 1122 frames with upper bodies marked

– First stage training (bootstrapping)• 1607 upper body annotations jittered to 32k positive samples• 55k negatives sampled from the same set of frames

– Second stage training (retraining)• 150k hard negatives found in the training data

Training data – positive annotations

Positive windows

Note: common size and alignment

Jittered positives

Jittered positives

Random negatives

Random negatives

Window (Image) first stage classification

HOG FeatureExtraction

Linear SVMClassifier

Jittered positives

random negatives f(x) = w>x+ b

• find high scoring false positives detections

• these are the hard negatives for the next round of training

• cost = # training images x inference on each image

Hard negatives

Hard negatives

First stage performance on validation set

Performance after retraining

Effects of retraining

Side by side

before retraining after retraining

Side by side

before retraining after retraining

Side by sidebefore retraining after retraining

Tracked upper body detections

Accelerating Sliding Window Search• Sliding window search is slow because so many windows are

needed e.g. x × y × scale ≈ 100,000 for a 320×240 image

• Most windows are clearly not the object class of interest

• Can we speed up the search?

Cascaded Classification• Build a sequence of classifiers with increasing complexity

ClassifierN

Face

Non-face

Classifier2

Non-face

Classifier1

Non-face

Window

More complex, slower, lower false positive rate

• Reject easy non-objects using simpler and faster classifiers

Possibly a face

Possibly a face

Cascaded Classification

• Slow expensive classifiers only applied to a few windows ⇒significant speed-up

• Controlling classifier complexity/speed:– Number of support vectors [Romdhani et al, 2001]– Number of features [Viola & Jones, 2001]– Type of SVM kernel [Vedaldi et al, 2009]

Summary: Sliding Window Detection• Can convert any image classifier into an

object detector by sliding window. Efficient search methods available.

• Requirements for invariance are reduced by searching over e.g. translation and scale

• Spatial correspondence can be “engineered in” by spatial tiling

Outline1. Image Classification

2. Object Category Detection

• Sliding window methods

• Histogram of Oriented Gradients (HOG)

• Learning an object detector

• PASCAL VOC (again) and two state of the art algorithms

3. The future and challenges

The PASCAL Visual Object Classes (VOC) Dataset and Challenge

Mark EveringhamLuc Van GoolChris Williams

John WinnAndrew Zisserman

Detection: Evaluation of Bounding Boxes• Area of Overlap (AO) Measure

Ground truth Bgt

Predicted Bp

Bgt � Bp

> ThresholdDetection if50%

• Evaluation: Average precision per class on predictions

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

recall

prec

isio

n

NLPR_HOGLBP_MC_LCEGCHLC (55.3)UOCTTI_LSVM_MDPM (54.3)UCI_DPM_SP (52.6)NUS_HOGLBP_CTX_CLS_RESCORE_V2 (52.4)MITUCLA_HIERARCHY (48.5)UVA_DETMONKEY (39.8)UVA_GROUPLOC (39.6)UMNECUIUC_HOGLBP_DHOGBOW_SVM (34.7)BONN_FGT_SEGM (33.7)UMNECUIUC_HOGLBP_LINSVM (33.7)CMU_RANDPARTS (31.7)LJKINPG_HOG_LBP_LTP_PLS2ROOTS (29.7)CMIC_SYNTHTRAIN (28.9)CMIC_VARPARTS (28.2)BONN_SVR_SEGM (24.4)TIT_SIFT_GMM_MKL2 (14.5)UC3M_GENDISC (5.5)TIT_SIFT_GMM_MKL (1.6)

Precision/Recall - Bicycle

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

recall

prec

isio

n

UOCTTI_LSVM_MDPM (49.1)NLPR_HOGLBP_MC_LCEGCHLC (46.7)UCI_DPM_SP (44.5)MITUCLA_HIERARCHY (43.5)UVA_GROUPLOC (37.8)UVA_DETMONKEY (36.9)UMNECUIUC_HOGLBP_DHOGBOW_SVM (33.8)UMNECUIUC_HOGLBP_LINSVM (33.1)BONN_SVR_SEGM (32.9)NUS_HOGLBP_CTX_CLS_RESCORE_V2 (32.8)BONN_FGT_SEGM (31.9)LJKINPG_HOG_LBP_LTP_PLS2ROOTS (27.5)CMU_RANDPARTS (19.5)CMIC_VARPARTS (13.7)CMIC_SYNTHTRAIN (13.3)TIT_SIFT_GMM_MKL2 (8.1)UC3M_GENDISC (5.4)TIT_SIFT_GMM_MKL (1.6)

Precision/Recall - Car

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

recall

prec

isio

n

UVA_GROUPLOC (13.0)UVA_DETMONKEY (12.1)UCI_DPM_SP (11.6)NLPR_HOGLBP_MC_LCEGCHLC (10.2)MITUCLA_HIERARCHY (9.7)NUS_HOGLBP_CTX_CLS_RESCORE_V2 (9.6)UOCTTI_LSVM_MDPM (9.1)UMNECUIUC_HOGLBP_LINSVM (7.2)BONN_SVR_SEGM (6.7)UMNECUIUC_HOGLBP_DHOGBOW_SVM (6.4)BONN_FGT_SEGM (5.8)LJKINPG_HOG_LBP_LTP_PLS2ROOTS (3.1)UC3M_GENDISC (1.5)CMU_RANDPARTS (1.1)TIT_SIFT_GMM_MKL2 (0.8)TIT_SIFT_GMM_MKL (0.3)

Precision/Recall – Potted plant

True Positives - MotorbikeMITUCLA_HIERARCHY

NLPR_HOGLBP_MC_LCEGCHLC

NUS_HOGLBP_CTX_CLS_RESCORE_V2

False Positives - MotorbikeMITUCLA_HIERARCHY

NLPR_HOGLBP_MC_LCEGCHLC

NUS_HOGLBP_CTX_CLS_RESCORE_V2

True Positives - CatUVA_DETMONKEY

UVA_GROUPLOC

MITUCLA_HIERARCHY

False Positives - CatUVA_DETMONKEY

UVA_GROUPLOC

MITUCLA_HIERARCHY

Progress 2008-2010

Results on 2008 data improve for best 2009 and 2010 methods for all categories, by over 100% for some categories

0

10

20

30

40

50

60ae

ropl

ane

bicy

cle bird

boat

bottl

e

bus

car

cat

chai

r

cow

dinin

gtab

le

dog

hors

e

moto

rbike

pers

on

potte

dpla

ntsh

eep

sofa

train

tvmo

nitor

Max

AP

(%)

200820092010

Multiple Kernels for Object Detection

Andrea Vedaldi, Varun Gulshan,Manik Varma, Andrew Zisserman

ICCV 2009

Approach

• Three stage cascade– Each stage uses a more powerful and more expensive classifier

• Multiple kernel learning for the classifiers over multiple features• Jumping window first stage

Feature vector

Fast Linear SVM

Quasi-linear SVM

Jumping Window

Non-linear SVM

Multiple Kernel Classification

PHOW Gray

Visual Words

PHOG

SSIM

PHOW Color

PHOG Sym

MK SVM

combine one kernel per histogram

[Varma & Rai, 2007][Gehler & Nowozin, 2009]

Feature vector

Jumping window

Hypothesis

Position of visual word with respect to the object

learn the position/scale/aspect ratio of the ROI with respect to the visual word

Trai

ning

Det

ectio

n

Handles change of aspect ratio

140

SVMs overview• First stage

– linear SVM– (or jumping window)– time: #windows

• Second stage– quasi-linear SVM– χ2 kernel– time: #windows × #dimensions

• Third stage– non-linear SVM– χ2-RBF kernel– time:

#windows × #dimensions × #SVs

140

Feature vector

Fast Linear SVM

Quasi-linear SVM

Jumping Window

Non-linear SVM

Results

Results

Results

Single Kernel vs. Multiple Kernels• Multiple Kernels gives substantial boost• Multiple Kernel Learning:

– small improvement over averaging– sparse feature selection

Object Detection with Discriminatively Trained Part Based Models

Pedro F. Felzenszwalb, David Mcallester, Deva Ramanan, Ross Girshick

PAMI 2010

Approach

• Mixture of deformable part-based models– One component per “aspect” e.g. front/side view

• Each component has global template + deformable parts• Discriminative training from bounding boxes alone

Example Model• One component of person model

root filterscoarse resolution

part filtersfiner resolution

deformationmodels

x1

x3

x4

x6

x5

x2

Object Hypothesis• Position of root + each part• Each part: HOG filter (at higher resolution)

Score is sum of filter scores minus

deformation costs

p0 : location of rootp1,..., pn : location of parts

z = (p0,..., pn)

Score of a Hypothesis

• Linear classifier applied to feature subset defined by hypothesis

filters deformation parameters

displacements

Appearance term Spatial prior

concatenation of HOG features and part displacement

features

concatenation of filters and deformation

parameters

Training• Training data = images + bounding boxes• Need to learn: model structure, filters, deformation costs

Training

Latent SVM (MI-SVM)

Minimize

Training data

We would like to find β such that:

Classifiers that score an example x using

β are model parametersz are latent values

• Which component?• Where are the parts?

SVM objective

Latent SVM Training

• Convex if we fix z for positive examples

• Optimization:– Initialize β and iterate:

• Pick best z for each positive example• Optimize β with z fixed

• Local minimum: needs good initialization– Parts initialized heuristically from root

Alternation strategy

Person Model

root filterscoarse resolution

part filtersfiner resolution

deformationmodels

Handles partial occlusion/truncation

Car Model

root filterscoarse resolution

part filtersfiner resolution

deformationmodels

Car Detections

high scoring false positiveshigh scoring true positives

Person Detections

high scoring true positiveshigh scoring false positives

(not enough overlap)

Comparison of Models

Summary• Multiple features and multiple kernels boost

performance• Discriminative learning of model with latent

variables for single feature (HOG):– Latent variables can learn best alignment in the

ROI training annotation– Parts can be thought of as local SIFT vectors– Some similarities to Implicit Shape

Model/Constellation models but with discriminative/careful training throughout

NB: Code available for latent model !

Outline

1. Image Classification

2. Object Category Detection

3. The future and challenges

Current Research Challenges• Context

– from scene properties: GIST, BoW, stuff – from other objects, e.g. Felzenszwalb et al, PAMI 10– from geometry of scene, e.g. Hoiem et al CVPR 06

• Occlusion/truncation– Winn & Shotton, Layout Consistent Random Field, CVPR 06– Vedaldi & Zisserman, NIPS 09– Yang et al, Layered Object Detection, CVPR 10

• 3D

• Scaling up – thousands of classes– Torralba et al, Feature sharing– ImageNet

• Weak and noisy supervision