![Page 1: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/1.jpg)
In Search of Art
Elliot J. Crowley and Andrew Zisserman
Visual Geometry Group
Department of Engineering Science
University of Oxford
![Page 2: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/2.jpg)
The Goal • An on-the-fly system for searching paintings
visually
• A user can type in the name of any category...
• Then hundreds of paintings containing that category will be retrieved in a matter of seconds
dog
![Page 3: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/3.jpg)
Benefits
• In many instances, the retrieved paintings will not have been known to contain the category
• Meaning these are new discoveries for the Art History community
dog
![Page 4: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/4.jpg)
Why is this good?
• Art historians can discover when something first appeared in paintings
• They can also observe how things have changed over time
![Page 5: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/5.jpg)
How is this achieved?
• Natural images annotated with object categories are everywhere.
• These can be used to learn object classifiers.
Google images of dog
![Page 6: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/6.jpg)
Dataset of Paintings
• We use `Your Paintings’ as the dataset
• `Your Paintings’ consists of over 210,000 paintings from UK galleries
http://www.bbc.co.uk/arts/yourpaintings/
• Method is independent of dataset however
• Can use other datasets e.g. Rijksmuseum or PrintART
![Page 7: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/7.jpg)
Outline
• Methodology
• Quantitative Evaluation
• Aligning retrieved objects
![Page 8: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/8.jpg)
What do we do?
• We crawl Google Images for a given category and learn a CNN-based classifier
• This classifier is applied to a dataset of paintings, retrieving paintings containing the category
![Page 9: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/9.jpg)
The Architecture
![Page 10: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/10.jpg)
How do we do this quickly?
• The bulk of the data has been pre-processed offline (negative training data, dataset of paintings)
• Online processing of Google Images is done in parallel across multiple cores
![Page 11: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/11.jpg)
In more detail…
• For a given query, the top 200 Google Image Hits are downloaded
• For each of these a CNN feature is computed online
• This is the positive training data
![Page 12: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/12.jpg)
Negative Training Data
• Offline, images are downloaded for Google searches of `things’ and ‘photos’
• The features for these are pre-computed
![Page 13: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/13.jpg)
Classification
• A Support Vector Machine is used to learn a classifier that discriminates the positive training data from the negative data
beard not beard
![Page 14: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/14.jpg)
Retrieval
• The classifier is applied to the pre-processed features of `Your Paintings’
• Each painting is given a score by the classifier
![Page 15: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/15.jpg)
Retrieval • The paintings are displayed in order of score.
beard
![Page 16: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/16.jpg)
The Architecture - Timings 0.5s
4.5s
<0.5s
<0.5s 2s
![Page 17: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/17.jpg)
Example Queries bridge
![Page 18: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/18.jpg)
Example Queries
carriage
![Page 19: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/19.jpg)
Example Queries
flower
![Page 20: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/20.jpg)
Example Queries
house
![Page 21: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/21.jpg)
Outline
• Methodology
• Quantitative Evaluation
• Aligning retrieved objects
![Page 22: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/22.jpg)
Quantitative Evaluation
• Evaluating the domain transfer problem of learning classifiers on natural images and applying these to paintings
![Page 23: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/23.jpg)
Test Set
• For this an annotated dataset of paintings is required
• 10,000 paintings in `Your Paintings’ have been tagged by the public
• These tags + painting titles are used to form the `Paintings Dataset’ with annotations corresponding to classes of PASCAL VOC
![Page 24: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/24.jpg)
The Paintings Dataset Class Paintings
with Class
Aeroplane 200
Bird 805
Boat 2143
Chair 1202
Cow 625
Dining-table 1201
Dog 1145
Horse 1493
Sheep 751
Train 329
• Assume complete annotation in the PASCAL sense
• Assess by calculating APs
Train
Dog
Horse
![Page 25: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/25.jpg)
Training Datasets
• 4 Datasets of natural images are used for training
• VOC12, VOC12+, Net Noisy, Net Curated
![Page 26: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/26.jpg)
Experiments
Features compared:
• Shallow Features - Fisher Vectors
VS.
• Deep Features - Convolutional Neural Networks (CNNs)
![Page 27: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/27.jpg)
Experiments - Features
• Fisher Vector VS. CNN Features
• CNN outperforms Fisher Vectors
• Added advantage of being lower dimensionality
![Page 28: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/28.jpg)
Augmentation • No augmentation
• C+F augmentation
224 224
224
256
![Page 29: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/29.jpg)
Experiments - Augmentation
• Sum Pool: Classifier applied to mean of augmented windows
• Max Pool: Classifier applied to each augmented window and maximum score recorded
• Best performance is aug + sum pool but almost as good with no aug + sum pool
![Page 30: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/30.jpg)
Experiments - Dimensionality
• 1K performs best
• Not that different from the others however
![Page 31: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/31.jpg)
Experiment Conclusions
• For the on-the-fly system 1K CNN features are used as these performed the best
• Sum pooled features are used for `Your Paintings’ as time is not a factor in computing these
• No augmentation is used on the images downloaded from Google (0.3s per image per core vs. 2.4s)
![Page 32: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/32.jpg)
Outline
• Methodology
• Quantitative Evaluation
• Aligning retrieved objects
![Page 33: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/33.jpg)
Alignment
• Some objects are automatically aligned…
moustache
![Page 34: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/34.jpg)
The Pencil Moustache
Anonymous Trendsetter, 1565
Copycats, Now
![Page 35: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/35.jpg)
Alignment
• Other objects require some work…
train
![Page 36: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/36.jpg)
Solution
Learn a DPM [1] on either
1. annotated bounding boxes (e.g. PASCAL VOC) or
2. the downloaded Google Images
[1] P Felzenszwalb, R Girshick, D McAllester, D Ramanan, Object Detection with Discriminatively Trained Part Based Models, CVPR 2010
![Page 37: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/37.jpg)
Auto-alignment
train
![Page 38: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/38.jpg)
Auto-alignment
horse
![Page 39: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/39.jpg)
Conclusion
• We provide a system that can find objects in paintings with high precision in very little time
• The objects found can be further curated using a DPM
![Page 40: In Search of Art - University of OxfordThe Goal •An on-the-fly system for searching paintings visually •A user can type in the name of any category... •Then hundreds of paintings](https://reader034.vdocuments.mx/reader034/viewer/2022051923/60110aa44c7f567d61542c27/html5/thumbnails/40.jpg)
Links
• VISOR: Visual Search of BBC News [1] http://www.robots.ox.ac.uk/~vgg/research/on-the-fly/ • CNN code [2] http://www.robots.ox.ac.uk/~vgg/research/deep_eval/ • Our system COMING SHORTLY! [1] K Chatfield, A Zisserman, VISOR: Towards On-the-Fly Large-Scale Object Category Retrieval, ACCV, 2012 [2] Ken Chatfield, Karen Simonyan, Andrea Vedaldi, Andrew Zisserman, Return of the Devil in the Details: Delving Deep into Convolutional Nets, BMVC, 2014