using visual features to improve tag suggestions in image sharing sites :: presentation @...
DESCRIPTION
TRANSCRIPT
Department for Information Technology, Klagenfurt University, Austria
Using Visual Features to Improve Tag Suggestions in Image Sharing Sites
Position of ..
o Mathias Lux, [email protected]
o Oge Marques, [email protected]
o Arthur Pitman, [email protected]
http://www.uni-klu.ac.at
2ITEC, Klagenfurt University, Austria
Agenda
● Motivation
● Proposed Architecture
● Current State
● Preliminary Conclusions
http://www.uni-klu.ac.at
3
Motivation
● 5,000 + uploads per minute on Flickr.
o Only 20%-25% are tagged
● Why are not all images tagged?
o Benefits of tagging are obvious …
o But effort is considered too high …
ITEC, Klagenfurt University, Austria
http://www.uni-klu.ac.at
4ITEC, Klagenfurt University, AustriaFocus on the annotation process …
http://www.uni-klu.ac.at
5
Motivation II
● Tagging images includes visual information
● Visual information retrieval in “narrow domains” has shown some success
o … to bridge the semantic gap
● Tags as narrow domains?
o e.g. Ferrari or sunset
ITEC, Klagenfurt University, Austria
http://www.uni-klu.ac.at
6
Assumptions & Process
● User has selected/uploaded a photo
● User has assigned at least one tag
● Our Task:
o Find more appropriate tags
o Present them to the user
o User decides which tags are “good”
ITEC, Klagenfurt University, Austria
http://www.uni-klu.ac.at
7
Our Approach
1. Find possible suggestions (tag based)
2. Find image sets per suggestion
3. Compare input image to different image sets
4. Re-rank the possible suggestions
ITEC, Klagenfurt University, Austria
http://www.uni-klu.ac.at
8
Example: Tag “juggling”
Input image
juggling + clown juggling + fire juggling + training
http://www.uni-klu.ac.at
9
Architecture
ITEC, Klagenfurt University, Austria
http://www.uni-klu.ac.at
10
Behind the curtains …
● Image sets are “ground truth” for tag suggestion
● Several (arbitrary) features extracted
● Fuzzy classifiers are trained
● Best feature+classifier is selected
● Input image gets classified
● Best matching class is ranked highest, etc.
ITEC, Klagenfurt University, Austria
http://www.uni-klu.ac.at
11
● Finding tag suggestions statistically
● Download image sets for suggestions
● Extract global image features
● Experiments with classifiers
http://www.uni-klu.ac.at
12
Current state
ITEC, Klagenfurt University, Austria
http://www.uni-klu.ac.at
13
Preliminary Conclusions
● Efficient implementation poses an engineering problem (CBIR, network, …)
● Promising results for some tags
o Found several tags considered as noise for our use case: flickrdiamonds, abigfave, 1imageaday, …
● We might find some “good questions” …
o How to define a “narrow domain”?
o How to find “narrow domains”?
o etc.
ITEC, Klagenfurt University, Austria
http://www.uni-klu.ac.at
14
Mathias Lux
Klagenfurt University, ITECAustria
Contact
o mlux @ itec.uni-klu.ac.at
o http://www.itec.uni-klu.ac.at/~mlux
o http://www.flickr.com/photos/mathias_l
ITEC, Klagenfurt University, Austria