improving web image search results using query-relative classifiers
DESCRIPTION
Improving web image search results using query-relative classifiers. Josip Krapacy Moray Allanyy Jakob Verbeeky Fr´ed´eric Jurieyy. Outline. Introduction Query-relative features Experimental evaluation Conclusion. Outline. Introduction Query-relative features Experimental evaluation - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/1.jpg)
Improving web image search results using query-relative classifiers
Josip Krapacy Moray Allanyy Jakob Verbeeky Fr´ed´eric Jurieyy
![Page 2: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/2.jpg)
OutlineIntroductionQuery-relative featuresExperimental evaluationConclusion
![Page 3: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/3.jpg)
OutlineIntroductionQuery-relative featuresExperimental evaluationConclusion
![Page 4: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/4.jpg)
IntroductionGoogle’s image search engine
have a precision of only 39%[16]Recently research improve image
search performance by visual information and not only text
Similar outlier detection, current setting the majority of retrieved image may be outliers, and inliers can be diverse
![Page 5: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/5.jpg)
IntroductionRecently methods have the same
drawback : ◦a separate image re-ranking model is
learned for each and every query – large number of possible queries make these approach wasted computational time
![Page 6: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/6.jpg)
IntroductionKey contribution :
◦Propose an image re-ranking method, based on textual and visual feature
◦Does not require learning a separate model for every query
◦The model parameters are shared across queries and learned once
![Page 7: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/7.jpg)
IntroductionImage re-ranking approach :
Our image re-ranking approach :
![Page 8: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/8.jpg)
OutlineIntroductionQuery-relative featuresExperimental evaluationConclusion
![Page 9: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/9.jpg)
Query-relative featuresQuery-relative text feature
◦Binary features◦Contextual features
Visual feature
Query-relative visual feature
![Page 10: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/10.jpg)
Query-relative text featureOur base query-relative text
feature follow [6,16]◦ContexR◦Context10◦Filedir◦Filename◦Imagealt◦Imagetitle◦Websitetitle
![Page 11: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/11.jpg)
Binary featureNine binary features indicate the
presence or absence of query terms :◦Surrounding text◦Image’s alternative text◦Web page’s title◦Image file’s URL’s hostname, directory
and filename◦Web page’s hostname, directory and
filenameWhich is active if some of the query
terms, but not all, are present in the field
![Page 12: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/12.jpg)
Contextual featuresCan be understood as a form of
pseudo-relevance feedback
Divide the image’s text annotation in three parts :◦Text surrounding the image◦Image’s alternative text◦Words in the web page’s title
![Page 13: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/13.jpg)
Contextual featuresDefine contextual features by
computing word histograms using all the image in the query set
Histogram of word counts : Image : iWord indexed : k
![Page 14: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/14.jpg)
Contextual featuresUse (1) to define a set of
additional context featuresThe kth binary feature represents
the presence or absence of kth most common word
We trim these features down to the first N element, so we have 9+9+3N binary feature
![Page 15: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/15.jpg)
Visual featuresOur image representation is based
on local appearance and position histograms
Local appearance◦Hierarchical k-means clustering ◦11-levels of quantisation, and k = 2
Position quantisation ◦Quad-tree with three level
The image is represented by appearance-position histogram
![Page 16: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/16.jpg)
Query-relative visual featuresNo direct correspondence
between query terms and image appearance
We can find which visual words are strongly associated with query set by contextual text features
Define a set of visual features to represent their presence or absence in a given image
![Page 17: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/17.jpg)
Query-relative visual featuresOrder the visual features :
◦A : query set◦T : training set◦ : average visual word
histogram
The kth feature relates to the visual word kth most related to this query
![Page 18: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/18.jpg)
Query-relative visual featuresWe compared three ways of
representing each visual word’s presence or absence◦The visual word’s normalised count
for this image ◦The ratio ◦Binary version of this ratio, threshold
at 1:
![Page 19: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/19.jpg)
OutlineIntroductionQuery-relative featuresExperimental evaluationConclusion
![Page 20: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/20.jpg)
Experimental evaluationNew data setModel trainingEvaluationRanking images by textual
featuresRanking images by visual
featuresCombining textual and visual
featuresPerformance on Fergus data set
![Page 21: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/21.jpg)
New data setPrevious data set contain image
for only a few classes, and at most case without their corresponding meta-data
In our data set, we provide the top-ranked images with their associated meta-data
Our data set of 353 image search queries and in total there are 71478 images
![Page 22: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/22.jpg)
Model trainingTrain a binary logistic
discriminant classifierQuery-relative features of
relevant images are used as positive examples
Query-relative features of irrelevant images are used as negative examples
Rank images for the query by the probability
Only need to be learnt once
![Page 23: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/23.jpg)
Evaluation Used mean average precisionLow Precision(LP): 25 queries where
the search engine performs worstHigh Precision(HP): 25 queries where
the search engine performs bestSearch Engine Poor(SEP): 25 queries
where the search engine least over random ordering of query set
Search Engine Good(SEG): 25 queries where the search engine most over random ordering of query set
![Page 24: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/24.jpg)
Ranking images by textual features
Diminishing gain per additional feature
![Page 25: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/25.jpg)
Ranking images by visual features
![Page 26: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/26.jpg)
Ranking images by visual features
Adding more visual features increases the overall performance, but with diminishing gain
![Page 27: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/27.jpg)
Combining textual and visual features
◦a = visual features, 50~400◦b = additional context features,
20~100
10%
![Page 28: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/28.jpg)
Performance on Fergus data set
Our method better than Google[4],[7] perform better, but they
require time-consuming training for every new query
![Page 29: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/29.jpg)
Results
![Page 30: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/30.jpg)
OutlineIntroductionQuery-relative featuresExperimental evaluationConclusion
![Page 31: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/31.jpg)
ConclusionConstruct query-relative features
that can be used to train generic classifiers
Rank images for previously unseen search queries without additional model training
The feature combined textual and visual information
Presence a new public data set
![Page 32: Improving web image search results using query-relative classifiers](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56815b81550346895dc97fbd/html5/thumbnails/32.jpg)
Thank you!!! &Happy New Year!!!!