Bundling interest points for object classification

Jordi Sánchez Escué

Supervised byXavier Giró i Nieto

Carles Ventura Royo

Contents● Introduction● State of the art● System Architecture

○ Feature extraction○ Classification○ Evaluation

● Experiments○ Class aggregation of interest points○ Bundling interest points○ Class aggregation & Bundling

● Conclusions & Future work1

Contents● Introduction● State of the art● System Architecture

○ Feature extraction○ Classification○ Evaluation

● Experiments○ Class aggregation of interest points○ Bundling interest points○ Class aggregation & Bundling

● Conclusions & Future work2

Introduction● Does this image contain a plane?

3

Introduction● Does this image contain a plane?

● Which type of flower is it?

4

Introduction● Mobile Visual Search

○ Generalist: Google Goggles

○ Leaf-based: Leafsnap

● Fine-grained classification○ Mushrooms

○ Flowers

5

Introduction● Textures around some interest points

6

Introduction● Features based on regions

7

Introduction● Explore combination: points & regions

8

Introduction● Project Requirements and Goals

○ Comparative study bundling interest points

9

Introduction● Project Requirements and Goals

○ Software Development

10

Contents● Introduction● State of the art● System Architecture

○ Feature extraction○ Classification○ Evaluation

● Experiments○ Class aggregation of interest points○ Bundling interest points○ Class aggregation & Bundling

● Conclusions & Future work11

State of the art● In Defense of Nearest-Neighbor Based

Image Classification, Oren Boiman

12

State of the art● Building contextual visual vocabulary for

large-scale image applications, S. Zhang

13

Contents● Introduction● State of the art● System Architecture

○ Feature extraction○ Classification○ Evaluation

● Experiments○ Class aggregation of interest points○ Bundling interest points○ Class aggregation & Bundling

● Conclusions & Future work14

System Architecture● Interest points and feature extraction

○ Sparse extraction

15

System Architecture● Interest points and feature extraction

○ Interest Points: SURF

16

System Architecture● Binary Partition Tree (BPT)

○ Partition: 20 reg. SLIC

17

● Binary Partition Tree○ A scale is chosen (ex, N = 3)

System Architecture

18

Contents● Introduction● State of the art● System Architecture

○ Feature extraction○ Classification○ Evaluation

● Experiments○ Class aggregation of interest points○ Bundling interest points○ Class aggregation & Bundling

● Conclusions & Future work19

System Architecture● Classification: Training

20

Trainer

1

2

3

4

System Architecture

CLASSIFIER1-NN, euclidean

distance

1

3

4

2

● Classification: Detection

21

Target image

System Architecture

Query image

22

System Architecture

Target image

Query image

23

System Architecture

Query image

Target image

24

System Architecture

Query image

Target image

25

System Architecture

Query image

Target image

26

System Architecture

Query image

Target image

27

System Architecture

Query image

Nearest Target image11

28

Contents● Introduction● State of the art● System Architecture

○ Feature extraction○ Classification○ Evaluation

● Experiments○ Class aggregation of interest points○ Bundling interest points○ Class aggregation & Bundling

● Conclusions & Future work29

System Architecture● Evaluation

○ Development of an evaluation tool

30

Tools

System Architecture● Software development

31

Trainer

Detector

Evaluation

SVM adapted to a flexible architecture

New tool for evaluation

Can be adapted to any classifier

or descriptor

Contents● Introduction● State of the art● System Architecture

○ Feature extraction○ Classification○ Evaluation

● Experiments○ Class aggregation of interest points○ Bundling interest points○ Class aggregation & Bundling

● Conclusions & Future work32

M-E. Nilsback & A. Zisserman, «A Visual Vocabulary for Flower Classification» Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2006. http://www.robots.ox.ac.uk/~vgg/data/flowers/17/

33

0.591769

0.3813720.463660

Experiments: basic approach● Results

34

Contents● Introduction● State of the art● System Architecture

○ Feature extraction○ Classification○ Evaluation

● Experiments○ Class aggregation of interest points○ Bundling interest points○ Class aggregation & Bundling

● Conclusions & Future work35

Experiments: Class aggregation● Aggregation of the interest points of all the

images of the same class to do the matching

36

Experiments: Class aggregation● Results

37

0.59

0.380.46

0.78

0.43

0.56

Contents● Introduction● State of the art● System Architecture

○ Feature extraction○ Classification○ Evaluation

● Experiments○ Class aggregation of interest points○ Bundling interest points○ Class aggregation & Bundling

● Conclusions & Future work38

● Region restriction

Experiments: Bundling interest points

39

Experiments: Bundling interest points

40

● Why the results did not improve?○ Image flower segmentation

Experiments: Bundling interest points

41

● Why the results did not improve?○ Bad flower segmentation (N = 2)

Experiments: Bundling interest points

42

● Why the results did not improve?○ Bad flower segmentation (N = 2)

● Future work to improve results○ Using perfect manual segmentation

Experiments: Bundling interest points

43

● Why the results did not improve?○ Good region matching (flower to flower)

Experiments: Bundling interest points

44

● Why the results did not improve?○ Bad region matching (flower to background)

Experiments: Bundling interest points

45

● Why the results did not improve?○ Bad region matching (flower to background)

● Future work to improve results○ Avoid using edge regions

○ Using object candidates

Experiments: Bundling interest points

46

Contents● Introduction● State of the art● System Architecture

○ Feature extraction○ Classification○ Evaluation

● Experiments○ Class aggregation of interest points○ Bundling interest points○ Class aggregation & Bundling

● Conclusions & Future work47

Experiments: Class aggregation & Bundling

● Class aggregation with points bundled in regions

48

● Comparative study

Experiments: Class aggregation & Bundling

49

Contents● Introduction● State of the art● System Architecture

○ Feature extraction○ Classification○ Evaluation

● Experiments○ Class aggregation of interest points○ Bundling interest points○ Class aggregation & Bundling

● Conclusions & Future work50

Conclusions & Future Work● Comparative study done

○ Bundling interest points into regions worsens the F1-score between 1% and 7%

○ Class aggregation improves the F1-score by 9.2%

● State of the art comparative study

○ Pointless having bad results

● Software development

● Future Work51

Bundling interest points for object classification

Jordi Sánchez Escué

Supervised byXavier Giró i Nieto

Carles Ventura Royo

System Architecture● Classification: Training

○ Semantic annotation & Ontology

......

...

...

...

...

1

2

3

4

System Architecture● Binary Partition Tree (BPT)

○ 20 SLIC superpixels

Future work● Add new approaches

○ Class aggregation in the query

○ Bundling query image, not bundling target

images (with certain spatial restriction).

● Optimize k, change classifier, more descriptors