convolutional features for instance search
TRANSCRIPT
Convolutional Features for Instance Search
Amaia Salvador
03/05/2016
2
Related Publications
E. Mohedano, A. Salvador, K. McGuinness, F. Marques, N. E. O'Connor and X. Giro, Bags of Local Convolutional Features for Scalable Instance SearchAccepted at ICMR 2016
A. Salvador, X. Giro, F. Marques, S. Satoh,Faster R-CNN Features for Instance SearchAccepted at DeepVision CVPRW 2016
Part IE. Mohedano, A. Salvador, K. McGuinness, F. Marques, N. E. O'Connor and X. Giro, Bags of Local Convolutional Features for Scalable Instance Search
Visual Image Retrieval
4Image Database
Visual Query
“A dog”
Expected outcome:
Visual Instance Retrieval
5Image Database
Visual Query
“This dog”
Expected outcome:
Visual Instance Retrieval
6
Image RepresentationsQuery image
Image Database
Image Matching Ranking List
Similarity score Image
...
0.98
0.97
0.10
0.01
v = (v1, …, vn)
v1 = (v11, …, v1n)
vk = (vk1, …, vkn)
...
Similarity Metric
(e.g. cosine similarity)
...
7
v1 = (v11, …, v1n)
vk = (vk1, …, vkn)...
INVERTED FILE
word Image ID1 1, 12, 2 1, 30, 1023 10, 124 2,3 6 10
...
Local hand-crafted features(e.g. SIFT)
Bag of Visual WordsN-Dimensional
feature space
Image Representations
High-dimensionalHighly sparse
8
Image Representations
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (pp. 1097-1105).
Convolutional Neural Networks
9
Image Representations
Babenko, A., Slesarev, A., Chigorin, A., & Lempitsky, V. (2014). Neural codes for image retrieval. In ECCV 2014Razavian, A., Azizpour, H., Sullivan, J., & Carlsson, S. (2014). CNN features off-the-shelf: an astounding baseline for recognition. In DeepVision CVPRW 2014
Convolutional Neural Networks FC layers as global feature representation
10
Image Representations
Babenko, A., & Lempitsky, V. (2015). Aggregating local deep features for image retrieval. ICCV 2015Tolias, G., Sicre, R., & Jégou, H. (2015). Particular object retrieval with integral max-pooling of CNN activations. ICLR 2015Kalantidis, Y., Mellina, C., & Osindero, S. (2015). Cross-dimensional Weighting for Aggregated Deep Convolutional Features. arXiv preprint arXiv:1512.04065.
Convolutional Neural Networks
sum/max pooled conv features as global representation
11
Image Representations
Ng, J., Yang, F., & Davis, L. (2015). Exploiting local features from deep networks for image retrieval. In DeepVision CVPRW 2015
Convolutional Neural Networks
conv features encoded with VLAD as global representation
12
MotivationDataset Complexity
TRECVID Instance Search464 hours of video content
13
Motivation: Image Representations
High-dimensional & Sparse
Bag of Visual Words
Compact & Dense(e.g. sum/max pooling conv feats, FC feats)
Capacity?
High-dimensional & Dense(e.g. VLAD encoding)
Scalability?
Methodology
15
Bag of Words Framework
16
Bag of Words Framework
(336x256)Resolution
conv5_1 from VGG16[1]
(42x32)
[1]Simonyan K., Zisserman A., Very Deep Convolutional Networks for Large-Scale Image Recognition, arXiv 2014
25K centroids 25K-D vector
17
Instance RetrievalQuery Representation
... ... ...
... ... ...
Global Search(GS)
Local Search(LS)
18
Spatial RerankingImage RepresentationsQuery image
Image Database
Image Matching Ranking List
v = (v1, …, vn)
v1 = (v11, …, v1n)
vk = (vk1, …, vkn)
...
Similarity Metric
(cosine similarity)
...
Top M imagesare locally analyzed
and reranked(M = 100)
19
Spatial Reranking
All window combinations with:
Query Image Target image in top M ranking
...
...
20
Query ExpansionImage RepresentationsQuery image
Image Database
Image Matching Ranking List
v = (v1, …, vn)
v1 = (v11, …, v1n)
vk = (vk1, …, vkn)
...
Similarity Metric
(cosine similarity)
...
Top N imagesare added to the query for a new
search(N = 5)
Experiments
22
DatasetsParis Buildings 6k Oxford Buildings 5k
TRECVID Instance Search 2013(subset of 23k frames)
Philbin, J. , Chum, O. , Isard, M. , Sivic, J. and Zisserman, A. Object retrieval with large vocabularies and fast spatial matching, CVPR 2007Philbin, J. , Chum, O. , Isard, M. , Sivic, J. and Zisserman, A. Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases. CVPR 2008Smeaton, A. F., Over, P., & Kraaij, W. Evaluation campaigns and TRECVid. ACM MM Multimedia information retrieval Workshop 2006
23
Results I: SoA Comparison
24
Results II: TRECVid INS
25
Qualitative Results
26
Conclusion
BoW encoding of convolutional features
• High-dimensional sparse representation suitable for fast retrieval
• Competitive results in two image retrieval benchmarks
• Well suited and more robust for scenarios where only small number of features are
in the target images are relevant to the query (INS).
Part IIA. Salvador, X. Giro, F. Marques, S. Satoh,Faster R-CNN Features for Instance Search
28
Reminder: Spatial Reranking
Query Image Target image in top M ranking
...
...
29
Reminder: Spatial Reranking
Koen E. A. van de Sande, Jasper R. R. Uijlings, Theo Gevers, Arnold W. M. Smeulders. Segmentation as Selective Search for Object Recognition, ICCV 2011
Object Proposals
30
Image & Region Representations
“dog”
CNN Architectures
plant, table, dog
CNN
CNN
Image Classification
Object Detection
31
Image & Region RepresentationsFaster R-CNN
Con
v la
yers
Region Proposal Network
FC6
Class probabilitiesFC7
FC8
RPN Proposals
RoI Pooling
Conv5_3
RPN Proposals
Ren, S., He, K., Girshick, R., & Sun, J. Faster R-CNN: Towards real-time object detection with region proposal networks. NIPS 2015
32
Image & Region RepresentationsFaster R-CNN
Con
v la
yers
Region Proposal Network
FC6
Class probabilitiesFC7
FC8
RPN Proposals
RoI Pooling
Conv5_3
RPN Proposals
Ren, S., He, K., Girshick, R., & Sun, J. Faster R-CNN: Towards real-time object detection with region proposal networks. NIPS 2015
Image representation
Region Representation
33
Image & Region Representations
Image representation Region Representation(for reranking)
RoI Pooling
Conv5_3 RoI Pooling
sum-pooling max-pooling
DD
34
Fine tuning for query objectsFaster R-CNN
Con
v la
yers
Region Proposal Network
FC6
Class probabilitiesFC7
FC8
RPN Proposals
RoI Pooling
Conv5_3
RPN Proposals
Train object detector for query instances using query images as training data
35
Fine tuning for query objectsFT #1: Train FC layers only
Con
v la
yers
Region Proposal Network
FC6
Class probabilitiesFC7
FC8
RPN Proposals
RoI Pooling
Conv5_3
RPN Proposals
36
Fine tuning for query objectsFT #2: Train all weights after conv2
Con
v la
yers
Region Proposal Network
FC6
Class probabilitiesFC7
FC8
RPN Proposals
RoI Pooling
Conv5_3
RPN Proposals
37
Spatial Reranking Strategies
Class-agnostic Spatial Reranking (CA-SR)
Query Image Database Image
FC6
Class probabilitiesFC7
FC8...
Class-specific Spatial Reranking (CS-SR)
38
Results
39Query image Top N retrieved images
40
ConclusionFaster R-CNN for Instance Search
• Suitable to obtain image and region features in a single forward pass
• Fine tuning as an effective solution to boost retrieval performance (subject to
application time constraints)
Con
v la
yers
Region Proposal Network
FC6
Class probabilitiesFC7
FC8
RPN Proposals
RoI Pooling
Conv5_3
RPN Proposals
Image representation
Region Representation
41
Thank you for your attention !
E. Mohedano, A. Salvador, K. McGuinness, F. Marques, N. E. O'Connor and X. Giro, Bags of Local Convolutional Features for Scalable Instance SearchAccepted at ICMR 2016
A. Salvador, X. Giro, F. Marques, S. Satoh,Faster R-CNN Features for Instance SearchAccepted at DeepVision CVPRW 2016