Информационный поиск, осень 2016: Федеративный поиск
TRANSCRIPT
![Page 1: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/1.jpg)
Distributed information retrieval Aggregated search Summary
Information RetrievalFederated Search
Ilya [email protected]
University of Amsterdam
Ilya Markov [email protected] Information Retrieval 1
![Page 2: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/2.jpg)
Distributed information retrieval Aggregated search Summary
Course overview
Data Acquisition
Data Processing
Data Storage
EvaluationRankingQuery Processing
Aggregated Search
Click Models
Present and Future of IR
Offline
Online
Advanced
Ilya Markov [email protected] Information Retrieval 2
![Page 3: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/3.jpg)
Distributed information retrieval Aggregated search Summary
Advanced topics in IR
Data Acquisition
Data Processing
Data Storage
EvaluationRankingQuery Processing
Aggregated Search
Click Models
Present and Future of IR
Offline
Online
Advanced
Ilya Markov [email protected] Information Retrieval 3
![Page 4: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/4.jpg)
Distributed information retrieval Aggregated search Summary
Outline
1 Distributed information retrieval
2 Aggregated search
3 Summary
Ilya Markov [email protected] Information Retrieval 4
![Page 5: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/5.jpg)
Distributed information retrieval Aggregated search Summary
Outline
1 Distributed information retrievalResource descriptionResource selectionScore normalization and results merging
2 Aggregated search
3 Summary
Ilya Markov [email protected] Information Retrieval 5
![Page 6: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/6.jpg)
Distributed information retrieval Aggregated search Summary
Taxonomy of IR architectures
Ilya Markov [email protected] Information Retrieval 6
![Page 7: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/7.jpg)
Distributed information retrieval Aggregated search Summary
Broker-based DIR
Q QR R
QR
...
BROKER
Q QR R
Q
...
BROKER
Q Q
Q
...
BROKER
Q
...
BROKER
...
BROKER
Q
...
BROKER
...
BROKER
1 Resource description
2 Resource selection
3 Score normalization
4 Results merging
Ilya Markov [email protected] Information Retrieval 7
![Page 8: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/8.jpg)
Distributed information retrieval Aggregated search Summary
Outline
1 Distributed information retrievalResource descriptionResource selectionScore normalization and results merging
Ilya Markov [email protected] Information Retrieval 8
![Page 9: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/9.jpg)
Distributed information retrieval Aggregated search Summary
Resource Description
Full content
Index of the full content
Metadata
Statistics: tf, df, average document length, . . .
Combination
Other. . .
Ilya Markov [email protected] Information Retrieval 9
![Page 10: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/10.jpg)
Distributed information retrieval Aggregated search Summary
DIR cooperation
Cooperative environments: a resource provides full access todocuments and indices and responds to queries
Uncooperative environments: a resource does not provide anyaccess to documents and indices; it only respond to queries
Ilya Markov [email protected] Information Retrieval 10
![Page 11: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/11.jpg)
Distributed information retrieval Aggregated search Summary
Query-based sampling
Q
DESCRIPTION
RESOURCE
Q
DESCRIPTION
RESOURCE
QRESOURCE
QRESOURCE
RESOURCE
J. Callan and M. Connell. Query-based sampling of text databases. ACM Transactions on Information Systems,19(2):97–130, 2001.
Ilya Markov [email protected] Information Retrieval 11
![Page 12: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/12.jpg)
Distributed information retrieval Aggregated search Summary
QBS issues
Q
DESCRIPTION
RESOURCE
DESCRIPTION
RESOURCE?
DESCRIPTION
RESOURCE?
?How to select queries for sampling?
When to stop sampling?
Ilya Markov [email protected] Information Retrieval 12
![Page 13: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/13.jpg)
Distributed information retrieval Aggregated search Summary
Selecting sampling queries: where from?
Other Resource Description (ORD): selects terms from areference dictionary
Learned Resource Description (LRD): selects terms from theretrieved documents based on term statistics
Ilya Markov [email protected] Information Retrieval 13
![Page 14: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/14.jpg)
Distributed information retrieval Aggregated search Summary
Selecting sampling queries: how?
Random selection
Document Frequency (df )
Collection Frequency (ctf )
Average Term Frequency (ctf /df )
Ilya Markov [email protected] Information Retrieval 14
![Page 15: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/15.jpg)
Distributed information retrieval Aggregated search Summary
Outline
1 Distributed information retrievalResource descriptionResource selectionScore normalization and results merging
Ilya Markov [email protected] Information Retrieval 15
![Page 16: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/16.jpg)
Distributed information retrieval Aggregated search Summary
Resource selection
1 Large-document model
2 Small-document model
3 Classification-based selection
Ilya Markov [email protected] Information Retrieval 16
![Page 17: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/17.jpg)
Distributed information retrieval Aggregated search Summary
Large-document model
QUERY
RESOURCESELECTION
0.19
0.870.61
0.46
0.08
...
0.87
0.61
0.46
0.19
0.08
QUERY
RESOURCESELECTION
0.19
0.870.61
0.46
0.08
...
0.87
0.61
0.46
0.19
0.08
QUERY
RESOURCESELECTION
0.19
0.870.61
0.46
0.08
QUERY
RESOURCESELECTION
QUERY
RESOURCESELECTION
1 Resources asdocuments
2 Retrieval ondocuments
3 Documents areresources
Ilya Markov [email protected] Information Retrieval 17
![Page 18: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/18.jpg)
Distributed information retrieval Aggregated search Summary
Large-document model examples
CORI: BM25 scoring function
LM-KL: Kullback-Leibler divergence between a query and aresource language models
Ilya Markov [email protected] Information Retrieval 18
![Page 19: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/19.jpg)
Distributed information retrieval Aggregated search Summary
Small-document model
1 Rank sample documents for a given user’s query
2 Consider the top n documents
3 Calculate a score for a resource R based on its documents inthe top-n
1 2 3 4 5 kk-1... ...
1 2 3 4 5 kk-1... ...top n
L. Si and J. Callan. Relevant document distribution estimation method for resource selection. In Proceedings ofSIGIR, pages 298–305, 2003.
Ilya Markov [email protected] Information Retrieval 19
![Page 20: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/20.jpg)
Distributed information retrieval Aggregated search Summary
Small-document model examples
1 2 3 4 5 kk-1... ...top n
Counts
score(Rred | q) = 3 · |Rred ||Sred |
Retrieval scores
score(Rgreen | q) = (score(d2 | q) + score(d7 | q)) · |Rgreen||Sgreen|
Ranks: n − rank(d |q) + 1
score(Ryellow | q) = (6 + 3 + 1) · |Ryellow ||Syellow |
= 10 · |Ryellow ||Syellow |
Ilya Markov [email protected] Information Retrieval 20
![Page 21: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/21.jpg)
Distributed information retrieval Aggregated search Summary
Classification-based selection
Classify each resource as “shown” or “not shown”
Use multiple features, including LD and SD
Ilya Markov [email protected] Information Retrieval 21
![Page 22: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/22.jpg)
Distributed information retrieval Aggregated search Summary
Outline
1 Distributed information retrievalResource descriptionResource selectionScore normalization and results merging
Ilya Markov [email protected] Information Retrieval 22
![Page 23: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/23.jpg)
Distributed information retrieval Aggregated search Summary
Score normalization and results merging
Score Normalization
UnsupervisedSemi-
supervised Supervised
Results MergingLinear Non-linear
Semi-supervisedUnsupervised
Score Normalization
UnsupervisedSemi-
supervised Supervised
Results MergingLinear Non-linear
Semi-supervisedUnsupervised
Ilya Markov [email protected] Information Retrieval 23
![Page 24: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/24.jpg)
Distributed information retrieval Aggregated search Summary
Linear score normalization
MinMax
snorm =s −min
max −min
Z-Score
snorm =s − µσ
Sum
s ′ = s −min, snorm =s ′∑i s′i
I. Markov, A. Arampatzis and F. Crestani. Unsupervised linear score normalization revisited. In Proceedings ofSIGIR, pages 1161–1162, 2012.
Ilya Markov [email protected] Information Retrieval 24
![Page 25: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/25.jpg)
Distributed information retrieval Aggregated search Summary
Score normalization and results merging
Score Normalization
UnsupervisedSemi-
supervised Supervised
Results MergingLinear Non-linear
Semi-supervisedUnsupervised
Ilya Markov [email protected] Information Retrieval 25
![Page 26: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/26.jpg)
Distributed information retrieval Aggregated search Summary
Results merging (CORI)
100
87
41
15
...
0.1
0.03
0.015
0.001
1
0.75
0.23
0.1
1250
815
800
234
... ... ...
100
87
41
15
...
0.1
0.03
0.015
0.001
1
0.75
0.23
0.1
1250
815
800
234
... ... ...
0.95 0.56 0.29 0.15
100
87
41
15
...
0.1
0.03
0.015
0.001
1
0.75
0.23
0.1
1250
815
800
234
... ... ...
1.38
1.2
0.57
0.2
...
1.22
0.37
0.18
0.12
1.12
0.84
0.27
0.11
1.06
0.69
0.68
0.2
... ... ...
0.95 0.56 0.29 0.15
snorm = sMinMax · (1 + λ · wresource)
J. Callan, Z. Lu and B.W. Croft. Searching distributed collections with inference networks. In Proceedings ofSIGIR, pages 21–28, 1995.
Ilya Markov [email protected] Information Retrieval 26
![Page 27: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/27.jpg)
Distributed information retrieval Aggregated search Summary
Distributed information retrieval summary
1 Resource description
Query-based sampling
2 Resource selection
Large-document modelSmall-document modelClassification-based selection
3 Score normalization and results merging
Linear/non-linearSupervised/semi-supervise/unsupervised
Ilya Markov [email protected] Information Retrieval 27
![Page 28: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/28.jpg)
Distributed information retrieval Aggregated search Summary
Outline
1 Distributed information retrieval
2 Aggregated searchVertical representationVertical selectionResults presentationSearch behaviorEvaluationPresent and future
3 Summary
Ilya Markov [email protected] Information Retrieval 28
![Page 29: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/29.jpg)
Distributed information retrieval Aggregated search Summary
Aggregated search
Ilya Markov [email protected] Information Retrieval 29
![Page 30: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/30.jpg)
Distributed information retrieval Aggregated search Summary
Aggregated search
1 Vertical Representation: represent each vertical as a set ofhigh-level features
2 Vertical Selection: given a query, decide for each vertical if ithas to be presented or not
3 Results Presentation: blend the results and position themappropriately on the page
Ilya Markov [email protected] Information Retrieval 30
![Page 31: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/31.jpg)
Distributed information retrieval Aggregated search Summary
Outline
2 Aggregated searchVertical representationVertical selectionResults presentationSearch behaviorEvaluationPresent and future
Ilya Markov [email protected] Information Retrieval 31
![Page 32: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/32.jpg)
Distributed information retrieval Aggregated search Summary
Vertical representation
Can be required for efficiency reasons
Vertical sampling : query-based sampling is used to retrievedocuments from each vertical. Sample queries are taken fromverticals’ query logs.
Wikipedia sampling : verticals are assigned to Wikipediacategories. Then documents are sampled from thesecategories.
J. Arguello, F. Diaz, J. Callan, and J.-F. Crespo. Sources of evidence for vertical selection. In Proceedings ofSIGIR, pages 315–322, 2009.
Ilya Markov [email protected] Information Retrieval 32
![Page 33: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/33.jpg)
Distributed information retrieval Aggregated search Summary
Outline
2 Aggregated searchVertical representationVertical selectionResults presentationSearch behaviorEvaluationPresent and future
Ilya Markov [email protected] Information Retrieval 33
![Page 34: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/34.jpg)
Distributed information retrieval Aggregated search Summary
Vertical selection
Classification-based selection
Features
Query string featuresQuery-log featuresCorpus features (LD, SD)
J. Arguello, F. Diaz, J. Callan, and J.-F. Crespo. Sources of evidence for vertical selection. In Proceedings ofSIGIR, pages 315–322, 2009.
Ilya Markov [email protected] Information Retrieval 34
![Page 35: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/35.jpg)
Distributed information retrieval Aggregated search Summary
Vertical Selection Results
Features Predictionprecision
diff %
all 0.583No query-logs 0.583 0.03%No triggers 0.583 -0.03%No query difficulty 0.582 -0.10%No geo-information 0.577† -1.01%No SD selection 0.568† -2.60%
Table: Multiple-feature predictors with one feature out, showing thecontribution of that feature
J. Arguello, F. Diaz, J. Callan, and J.-F. Crespo. Sources of evidence for vertical selection. In Proceedings ofSIGIR, pages 315–322, 2009.
Ilya Markov [email protected] Information Retrieval 35
![Page 36: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/36.jpg)
Distributed information retrieval Aggregated search Summary
Outline
2 Aggregated searchVertical representationVertical selectionResults presentationSearch behaviorEvaluationPresent and future
Ilya Markov [email protected] Information Retrieval 36
![Page 37: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/37.jpg)
Distributed information retrieval Aggregated search Summary
Results presentation
Learning to rank
Pre-retrieval features
Query: named-entity typeVertical: categoryQuery-vertical: click-through rateQuery-vertical: vertical intent
Post-retrieval features
Hit countsText similarity
Figure 3: A screenshot of the SERP showing results from ver-ticals at ToP, MoP and BoP.where x1 and x2 are the feature vectors for the queries q1 and q2
respectively. To find this function !V , we generate a set of pairs!x, l" that we call examples. Then we try to find a function !V thatbest fits the data. For this, we use a learning algorithm, which isdescribed in Section 5.
In an example !x, l", l is referred to as a label. In some sense, thel is the value we want the function !V to predict for input x. Wecan provide contradicting examples too. In this case, the learningalgorithm tries to find !V that best fits these examples under somemetric. We may also associate a weight w with each example !x, l"that indicates the relative importance of the example. So our inputto the learning algorithm is a list of tuples !x, l, w" There are twocommon methods for generating the examples for training:
• Sample a set of queries for which the vertical triggers. Thenshow the contents of the web results and the vertical to judgesand ask them to assign a label, say 0 or 1 for relevant or notrelevant, or a number in some range like 0, 1, 2, 3 indicatingthe degree of the relevance of the vertical for the query.
• Sample a set of queries for which the vertical triggers fromthe user logs, and based on the user behavior, assign a label,say 0 or 1 to indicate if the vertical may have satisfied theuser’s information need.
In this paper, we concentrate on the latter approach using the formeronly for comparison in our offline analysis. The details of our labelgeneration process follow in Section 5.3.
4.2 Calibration and PlacementOnce we arrive at a ranking function rV , our next task is to match
coverages at ToP, MoP and BoP based on our agreements with ver-tical owners. This is a straight-forward process. Given a sampleQ of queries for which the vertical triggers, we compute the valueprovided by the ranking function !V for these queries, sort themand pick two thresholds !V,ToP and !V,MoP such that
|{q # Q : !V ("(q, Wq, vq)) $ !V,ToP }|/|Q| = cToP
and
|{q # Q : !V,ToP > !V ("(q, Wq, vq)) $ !V,MoP }|/|Q| = cMoP .
This gives us a function µV : R %& S where
µV (x) =
(ToP if x $ !V,ToP
MoP if !V,ToP > x $ !V,MoP
BoP otherwise.
Finally, the function fV that we wanted to compute can be writ-ten as
fV (q, Wq, vq) = µV (!V ("V (q, Wq, vq))).
For convenience, with a slight abuse of notation, we will drop thevertical V from the subscript (since we rank each vertical individ-ually) and we may use !(q) to denote !V ("V (q, Wq, vq)).
5. STATISTICAL MODELINGIn this section, we first describe Gradient Boosted Decision Trees
(GBDT), which is the machine learning algorithm we used to learnonline user preference and predict human relevance grades. Thenwe describe how we represent the data using features such that theweb component of the feature vector becomes a reference frameor anchor, and finally we describe how we generate training labelsfrom online user click logs.
5.1 Gradient Boosted Decision TreesGradient boosted decision trees (GBDT) are non-parametric re-
gression models [8]. GBDTs and its stochastic variant [9] computea function approximation by performing a numerical optimizationin the function space instead of the parameter space. We providean overview of the GBDT algorithm.
A basic regression tree f(x), x # RK , partitions the space of ex-planatory variable values into disjoint regions Rj , j = 1, 2, . . . , Jassociated with the terminal nodes of the tree. Each region is as-signed a value "j such that f(x) = "j if x # Rj . Thus the com-plete tree is represented as:
T (x;!) =JX
j=1
"jI(x # Rj),
where ! = {Rj , "j}J1 , and I is the indicator function. Let the
given training data be denoted by (xi, yi, wi), i = 1, . . . , N . Thatis, xi are the observed feature vectors, yi the labels, and wi theweight associated with the training pair (xi, yi). For a loss functionL(yi, "j), parameters are estimated by minimizing the total loss:
!̂ = arg min!
JX
j=1
X
xi!Rj
wi · L(yi, "j).
In our experiments, we perform regression using the squared erroras loss function.
A boosted tree is an aggregate of such trees, each of which iscomputed in a sequence of stages. That is,
fM (x) =MX
m=1
T (x;!m),
718
A. K. Ponnuswami, K. Pattabiraman, Q. Wu, R. Gilad-Bachrach, and T. Kanungo. On composition of a federatedweb search result page: using online users to provide pairwise preference for heterogeneous verticals. InProceedings of WSDM, pages 715–724, 2011.
Ilya Markov [email protected] Information Retrieval 37
![Page 38: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/38.jpg)
Distributed information retrieval Aggregated search Summary
Outline
2 Aggregated searchVertical representationVertical selectionResults presentationSearch behaviorEvaluationPresent and future
Ilya Markov [email protected] Information Retrieval 38
![Page 39: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/39.jpg)
Distributed information retrieval Aggregated search Summary
Search behavior: eye fixation
None of the changes were detectable by the subjects. While being asked after their query sessions, none of the subjects suspect any manipulation. This server is also used to log all click-through behavior and all SERPs subjects visit.
All subjects’ eye movements are recorded using a SMI RED250 eye-tracker, which utilizes infrared to reconstruct the subjects’ eye position. BeGaze Experimental Center is used for the simultaneous acquisition and analysis of the subjects’ eye movements. With this tracking device, the following indicators of ocular behaviors are recorded: fixations, saccades, pupil dilation, and scan paths [12]. Among these behaviors, we focus on eye fixation, which is the most relevant metric for evaluating information processing in online search. In this paper, eye fixation is defined as a spatially stable gaze lasting for approximately 200-300 milliseconds, during which visual attention is directed to a specific area of the visual display.
4.2 Do Users Examine Verticals First? To analyze which result users first pay attention to, we collect
subjects’ first two seconds eye fixations on the screen. Figure 8 shows two samples from eye-tracking data which shows users’ watching area on SERP with different kinds of verticals or no vertical results. From this figure we can see that users pay most attention to the first result when there is no vertical in SERP (which should be regarded as sign for position bias). However, when there is a multimedia vertical result at the third position, it attracts a lot of users’ direct attentions.
We set 250 milliseconds as the threshold of fixation action and labeled each document’s boundary manually. Then we can record each subject’s eye examining sequence on document results for each SERP. This statistical result shows users’ examining sequential behaviors. We compared users’ first examining behavior for each vertical class using the same form with users’ first click distribution we analyzed in previous section. Figure 9 shows the subjects’ first examining distribution at each document position for each vertical class compared with no vertical situation. From Figure 9(b) we can see that when multimedia vertical is placed on the first screen (rank 1, 3 and 5); it actually attracts more attention than ordinary results. This conclusion is consistent with CT 2 in section 3.5. For application verticals, Figure 9(c) shows that application vertical attracts slightly more attention than ordinary result. Meanwhile Figure 9(a) shows that text vertical doesn’t attract much user attention.
4.3 Behavior after Examining Verticals First To validate the findings in Section 3 about how users behave after clicking a vertical results first. We look into the examining sequences of subjects when they examine vertical results firstly.
Two typical examining sequential patterns are extracted from eye-tracking data and shown in Figure 10. We can see from this figure that users examine results above verticals either sequentially (top-down) or from the ones next to verticals first (bottom-up). Further statistics in Table 2 show that most of the subjects examine back to top results after examining a vertical first. This finding accords with our assumption in Section 3.3 that users will resume a top-down search session after being interrupted by a vertical result.
(a)
(b)
Figure 8. Heat map of the subjects’ eye fixation areas in first 2 seconds on (a) SERP with no vertical (b) SERP with
multimedia vertical placed at the 3rd position
(a) (b) (c)Figure 9. First examining distribution of SERPs with (a) text (b) multimedia (c) application vertical when vertical result is placed at rank 1, 3, 5 and 10 compared with SERPs with no verticals. (Brighter color means a higher first examining rate on document position. We don’t show document positions from 6 to 10 here because almost no subjects examine results located on them first)
508
None of the changes were detectable by the subjects. While being asked after their query sessions, none of the subjects suspect any manipulation. This server is also used to log all click-through behavior and all SERPs subjects visit.
All subjects’ eye movements are recorded using a SMI RED250 eye-tracker, which utilizes infrared to reconstruct the subjects’ eye position. BeGaze Experimental Center is used for the simultaneous acquisition and analysis of the subjects’ eye movements. With this tracking device, the following indicators of ocular behaviors are recorded: fixations, saccades, pupil dilation, and scan paths [12]. Among these behaviors, we focus on eye fixation, which is the most relevant metric for evaluating information processing in online search. In this paper, eye fixation is defined as a spatially stable gaze lasting for approximately 200-300 milliseconds, during which visual attention is directed to a specific area of the visual display.
4.2 Do Users Examine Verticals First? To analyze which result users first pay attention to, we collect
subjects’ first two seconds eye fixations on the screen. Figure 8 shows two samples from eye-tracking data which shows users’ watching area on SERP with different kinds of verticals or no vertical results. From this figure we can see that users pay most attention to the first result when there is no vertical in SERP (which should be regarded as sign for position bias). However, when there is a multimedia vertical result at the third position, it attracts a lot of users’ direct attentions.
We set 250 milliseconds as the threshold of fixation action and labeled each document’s boundary manually. Then we can record each subject’s eye examining sequence on document results for each SERP. This statistical result shows users’ examining sequential behaviors. We compared users’ first examining behavior for each vertical class using the same form with users’ first click distribution we analyzed in previous section. Figure 9 shows the subjects’ first examining distribution at each document position for each vertical class compared with no vertical situation. From Figure 9(b) we can see that when multimedia vertical is placed on the first screen (rank 1, 3 and 5); it actually attracts more attention than ordinary results. This conclusion is consistent with CT 2 in section 3.5. For application verticals, Figure 9(c) shows that application vertical attracts slightly more attention than ordinary result. Meanwhile Figure 9(a) shows that text vertical doesn’t attract much user attention.
4.3 Behavior after Examining Verticals First To validate the findings in Section 3 about how users behave after clicking a vertical results first. We look into the examining sequences of subjects when they examine vertical results firstly.
Two typical examining sequential patterns are extracted from eye-tracking data and shown in Figure 10. We can see from this figure that users examine results above verticals either sequentially (top-down) or from the ones next to verticals first (bottom-up). Further statistics in Table 2 show that most of the subjects examine back to top results after examining a vertical first. This finding accords with our assumption in Section 3.3 that users will resume a top-down search session after being interrupted by a vertical result.
(a)
(b)
Figure 8. Heat map of the subjects’ eye fixation areas in first 2 seconds on (a) SERP with no vertical (b) SERP with
multimedia vertical placed at the 3rd position
(a) (b) (c)Figure 9. First examining distribution of SERPs with (a) text (b) multimedia (c) application vertical when vertical result is placed at rank 1, 3, 5 and 10 compared with SERPs with no verticals. (Brighter color means a higher first examining rate on document position. We don’t show document positions from 6 to 10 here because almost no subjects examine results located on them first)
508
Wang et al. Incorporating vertical results into search click models. In Proceedings of SIGIR, pages 503–512,2013.
Ilya Markov [email protected] Information Retrieval 39
![Page 40: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/40.jpg)
Distributed information retrieval Aggregated search Summary
Search behavior: examination order
(a)
(b)
Figure 10. Typical eye-tracking cases of examining previous document (a) bottom up (b) top down after first examining
verticals. The examining sequence is from green circles, yellow circles (dashed border) to red circles (solid border)
Table 2 shows the proportion of each examining patterns when user first examines verticals. We can see that after examining vertical result first, most subjects (89% users who examine 3rd result first and 100% users who examine 5th result first) scan back to the previous results. This shows that users may be attracted by the vertical’s presentation and change their examination sequence; meanwhile results on top of the ranking list are always valued and not omitted by users.
Table 2. Proportion of different examining behaviors after user first examines a vertical result
Vertical Rank #Subject1 Next Previous Back to Top 3 9 0.11 0.22 0.67
5 2 0.00 0.00 1.00
4.4 Eye-tracking Analysis Findings In summary, we can conclude the main influence of vertical results on users’ examining behavior into two aspects: ET 1. Multimedia and application vertical results are examined
more frequently compared with ordinary web results. ET 2. After examining a vertical result first, most users will scan
back to examine the previous results before the vertical either bottom-up or top-down.
1 11 subjects out of all 22 subjects examine vertical result first
when vertical is placed at the 3rd or 5th position. We don’t consider sessions in which verticals are placed at the 1st position because there would be no “previous” or “back to top” patterns.
5. VERTICAL-AWARE CLICK MODEL We first state some definitions and notations that will be used
in the following part. A search session within the same query is called a query session. A web search user initializes a query session s by submitting a query q to the search engine. The SERP can be represented as sequentially (M = 10 if we only consider the first search result page), where is document at position from the top of the page.
Examination, click and document relevance are treated as probabilistic events. In particular, for a given query session, we use binary random variables , and to represent the examination, click and document attractiveness events of the document at position . The corresponding, examination and click probabilities for position are denoted by , and , respectively.
5.1 Preliminaries We first introduce two important hypotheses: examination
hypothesis and cascade hypothesis, which are the foundations of most existing click models.
The examination hypothesis [13] can be summarized as follows: | (1) | (2)
where . is defined as the document relevance, which is the conditional probability of a click event after examination. Given , is conditionally independent on previous examine/click events.
The cascade hypothesis in [8] assumes that users always begin the examination at the first document. The examination is strictly linear from top to bottom of the search result page, so a document is examined only if all previous documents are examined:
(3) | (4)
Given , is conditionally independent of all examine/click events above , but may depend on the click .
The user browsing model (UBM) [3] is based on the examination hypothesis, but it doesn’t follow the cascade hypothesis. Instead, it assumes that the examination probability depends on its own position and the previous clicked position :
| (5)
Given click , is conditionally independent of all previous examination events . If there is no click before , is set to 0. The probability of a query session under UBM is:
∏ (6)
5.2 Modeling Biases We can see from Section 3 and 4 that users treat verticals
differently from ordinary results. Now, we want to develop an effective click model for federated search containing both vertical and ordinary results. Notice that we only consider the situation that only one vertical appears in the SERP. Therefore, if there are two or more verticals in SERP, we only keep the first vertical and simply regard others as ordinary results. 5.2.1 Attraction Bias
According to the conclusions of ET 1 and CT 3 described in previous sections, certain vertical result (e.g. multimedia vertical) will attract users’ attention directly and cause users to examine and thus click it first. So we formalize the assumption as: Assumption 1 (Attraction Bias): If there is a vertical placed in the SERP, there is probability that users examine it first.
509
(a)
(b)
Figure 10. Typical eye-tracking cases of examining previous document (a) bottom up (b) top down after first examining
verticals. The examining sequence is from green circles, yellow circles (dashed border) to red circles (solid border)
Table 2 shows the proportion of each examining patterns when user first examines verticals. We can see that after examining vertical result first, most subjects (89% users who examine 3rd result first and 100% users who examine 5th result first) scan back to the previous results. This shows that users may be attracted by the vertical’s presentation and change their examination sequence; meanwhile results on top of the ranking list are always valued and not omitted by users.
Table 2. Proportion of different examining behaviors after user first examines a vertical result
Vertical Rank #Subject1 Next Previous Back to Top 3 9 0.11 0.22 0.67
5 2 0.00 0.00 1.00
4.4 Eye-tracking Analysis Findings In summary, we can conclude the main influence of vertical results on users’ examining behavior into two aspects: ET 1. Multimedia and application vertical results are examined
more frequently compared with ordinary web results. ET 2. After examining a vertical result first, most users will scan
back to examine the previous results before the vertical either bottom-up or top-down.
1 11 subjects out of all 22 subjects examine vertical result first
when vertical is placed at the 3rd or 5th position. We don’t consider sessions in which verticals are placed at the 1st position because there would be no “previous” or “back to top” patterns.
5. VERTICAL-AWARE CLICK MODEL We first state some definitions and notations that will be used
in the following part. A search session within the same query is called a query session. A web search user initializes a query session s by submitting a query q to the search engine. The SERP can be represented as sequentially (M = 10 if we only consider the first search result page), where is document at position from the top of the page.
Examination, click and document relevance are treated as probabilistic events. In particular, for a given query session, we use binary random variables , and to represent the examination, click and document attractiveness events of the document at position . The corresponding, examination and click probabilities for position are denoted by , and , respectively.
5.1 Preliminaries We first introduce two important hypotheses: examination
hypothesis and cascade hypothesis, which are the foundations of most existing click models.
The examination hypothesis [13] can be summarized as follows: | (1) | (2)
where . is defined as the document relevance, which is the conditional probability of a click event after examination. Given , is conditionally independent on previous examine/click events.
The cascade hypothesis in [8] assumes that users always begin the examination at the first document. The examination is strictly linear from top to bottom of the search result page, so a document is examined only if all previous documents are examined:
(3) | (4)
Given , is conditionally independent of all examine/click events above , but may depend on the click .
The user browsing model (UBM) [3] is based on the examination hypothesis, but it doesn’t follow the cascade hypothesis. Instead, it assumes that the examination probability depends on its own position and the previous clicked position :
| (5)
Given click , is conditionally independent of all previous examination events . If there is no click before , is set to 0. The probability of a query session under UBM is:
∏ (6)
5.2 Modeling Biases We can see from Section 3 and 4 that users treat verticals
differently from ordinary results. Now, we want to develop an effective click model for federated search containing both vertical and ordinary results. Notice that we only consider the situation that only one vertical appears in the SERP. Therefore, if there are two or more verticals in SERP, we only keep the first vertical and simply regard others as ordinary results. 5.2.1 Attraction Bias
According to the conclusions of ET 1 and CT 3 described in previous sections, certain vertical result (e.g. multimedia vertical) will attract users’ attention directly and cause users to examine and thus click it first. So we formalize the assumption as: Assumption 1 (Attraction Bias): If there is a vertical placed in the SERP, there is probability that users examine it first.
509
Wang et al. Incorporating vertical results into search click models. In Proceedings of SIGIR, pages 503–512,2013.
Ilya Markov [email protected] Information Retrieval 40
![Page 41: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/41.jpg)
Distributed information retrieval Aggregated search Summary
Results coherence
Top results in web search are more diversified than top resultsin vertical search
Images vertical has a stronger spillover effect than news,shopping, and video verticals
Top-positioned vertical causes a stronger spillover effect thanright-positioned vertical
J. Arguello and R. Capra. The Effects of Aggregated Search Coherence on Search Behavior. ACM Transactionson Information Systems, 35(1), 2:1–2:30, 2016.
Ilya Markov [email protected] Information Retrieval 41
![Page 42: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/42.jpg)
Distributed information retrieval Aggregated search Summary
Outline
2 Aggregated searchVertical representationVertical selectionResults presentationSearch behaviorEvaluationPresent and future
Ilya Markov [email protected] Information Retrieval 42
![Page 43: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/43.jpg)
Distributed information retrieval Aggregated search Summary
Evaluation
TREC FedWeb 2013–2014
Evaluation framework by Zhou et al. (SIGIR’12)
Util(page) =
∑Bi=1 Exam(blocki ) · Gain(blocki )∑Bi=1 Exam(blocki ) · Effort(blocki )
Ilya Markov [email protected] Information Retrieval 43
![Page 44: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/44.jpg)
Distributed information retrieval Aggregated search Summary
Outline
2 Aggregated searchVertical representationVertical selectionResults presentationSearch behaviorEvaluationPresent and future
Ilya Markov [email protected] Information Retrieval 44
![Page 45: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/45.jpg)
Distributed information retrieval Aggregated search Summary
Direct answers and cards
Picture taken from https://www.rezstream.com/blog/predictive-search-through-google-now
Ilya Markov [email protected] Information Retrieval 45
![Page 46: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/46.jpg)
Distributed information retrieval Aggregated search Summary
Direct answers and cards
Proactive vs. reactive
Observations
Top-heavy click distributionUsage is affected by temporaland local dynamics
Re-ranking proactive cards
Observations → featuresClicks and viewport duration →relevance
From Queries to Cards
Re-ranking Proactive Card Recommendations Based on Reactive Search History
Milad ShokouhiMicrosoft
Cambridge, United [email protected]
Qi GuoMicrosoft
Bellevue, [email protected]
General TermsThe growing accessibility of mobile devices has substantially re-formed the way users access information. While the reactive searchby query remains as common as before, recent years have witnessedthe emergence of various proactive systems such as Google Nowand Microsoft Cortana. In these systems, relevant content is pre-sented to users based on their context without a query. Interestingly,despite the increasing popularity of such services, there is very littleknown about how users interact with them.
In this paper, we present the first study on user interactions withinformation cards. We demonstrate that the usage patterns of thesecards vary depending on time and location. We also show that whileoverall different topics are clicked by users on proactive and reactiveplatforms, the topics of the clicked documents by the same usertend to be consistent cross-platform. Furthermore, we propose asupervised framework for re-ranking proactive cards based on theuser’s context and past history. To train our models, we use theviewport duration and clicks to infer pseudo-relevance labels for thecards. Our results suggest that the quality of card ranking can besignificantly improved particularly when the user’s reactive searchhistory is matched against the proactive data about the cards.
Categories and Subject DescriptorsInformation Systems [Information Retrieval]: Users and interac-tive retrieval—Personalization
KeywordsProactive ranking; zero query; Information cards
1. INTRODUCTIONMobile devices account for a significant fraction of online search
and browsing traffic. The number of mobile queries has grownfivefold in the past three years [6]. In fact, the number of mobilequeries has recently exceeded the number of those submitted fromdesktop devices in the United States and many other countries [1].Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bearthis notice and the full citation on the first page. Copyrights for componentsof this work owned by others than ACM must be honored. Abstracting withcredit is permitted. To copy otherwise, or republish, to post on servers or toredistribute to lists, requires prior specific permission and/or a fee. Requestpermissions from [email protected]’15, August 09 - 13, 2015, Santiago, Chile.c� 2015 ACM. ISBN 978-1-4503-3621-5/15/08 ...$15.00.
DOI: http://dx.doi.org/10.1145/2766462.2767705.
Figure 1: Examples of interfaces with proactive “cards” presentedrespectively in Apple Siri, Google Now and Microsoft Cortana (top).Examples of reactive search queries submitted to Google on mobileand Bing on desktop (bottom) for query paris shooting.
Consequently, information retrieval systems are constantly evolvinginto more contextual and mobile-friendly services.
Among recent enhancements, the introduction of zero-query sys-tems [8] is arguably one of the most significant breakthroughs. In2012 at a workshop, a group of 45 leading experts in the field ac-knowledged zero-query search to be one of the six most interestingresearch directions in information retrieval (IR).
“Future information retrieval systems must anticipateuser needs and respond with information appropriate tothe current context without the user having to enter aquery.” [8]
The major players in the search industry have embraced the newtrends by releasing systems such as Google Now [4], Apple Siri[5] and Microsoft Cortana [2]. In all these systems, in addition tothe typical reactive search by a query, proactive information cardsare presented to users based on their context. Proactive systems are
695
M. Shokouhi and Q. Guo. From Queries to Cards: Re-ranking Proactive Card Recommendations Based onReactive Search History. In Proceedings of SIGIR, pages 695–704, 2015.
Ilya Markov [email protected] Information Retrieval 46
![Page 47: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/47.jpg)
Distributed information retrieval Aggregated search Summary
Whole-page relevance
Ilya Markov [email protected] Information Retrieval 47
![Page 48: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/48.jpg)
Distributed information retrieval Aggregated search Summary
Whole-page relevance
1 Page content x
2 Page presentation p
3 SERP (x,p)
4 User response y
5 User satisfaction s = g(y)
Find the presentation p for a given page content x,such that when the SERP (x,p) is presented to the user,
her satisfaction score s is maximized.
Wang et al. Beyond Ranking: Optimizing Whole-Page Presentation. In Proceedings of WSDM, pages 103–112,2016.
Ilya Markov [email protected] Information Retrieval 48
![Page 49: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/49.jpg)
Distributed information retrieval Aggregated search Summary
Whole-page relevance
1 Page content xquery featuressearch result featurescorpus-level and global result features
2 Page presentation pbinary indicator: “shown”/“not shown”categorical features (e.g., multimedia type)numerical features (e.g., brightness and contrast)
3 SERP (x,p)4 User response y
clickscollected through a randomization experiment
5 User satisfaction s = g(y)click-skip metric: g(y) =
∑ki=1 yi , where yi ∈ {−1, 1}
Wang et al. Beyond Ranking: Optimizing Whole-Page Presentation. In Proceedings of WSDM, pages 103–112,2016.
Ilya Markov [email protected] Information Retrieval 49
![Page 50: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/50.jpg)
Distributed information retrieval Aggregated search Summary
Constructing wikipedia-like pages from search results
Ilya Markov [email protected] Information Retrieval 50
![Page 51: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/51.jpg)
Distributed information retrieval Aggregated search Summary
Aggregated search summary
1 Vertical representation using sampling
2 Vertical selection as classification
3 Results presentation as learning to rank
4 Biases in search behavior
5 Evaluation6 Present and future
Direct answers and cardsWhole-page relevanceConstructing wikipedia-like pages from search results
Ilya Markov [email protected] Information Retrieval 51
![Page 52: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/52.jpg)
Distributed information retrieval Aggregated search Summary
Outline
1 Distributed information retrieval
2 Aggregated search
3 Summary
Ilya Markov [email protected] Information Retrieval 52
![Page 53: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/53.jpg)
Distributed information retrieval Aggregated search Summary
Federated search summary
Distributed information retrieval
Resource descriptionResource selectionScore normalization and results merging
Aggregated search
Vertical representationVertical selectionResults presentation
Ilya Markov [email protected] Information Retrieval 53
![Page 54: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/54.jpg)
Distributed information retrieval Aggregated search Summary
Materials
Milad Shokouhi and Luo SiFederated SearchFoundations and Trends in Information Retrieval, 2011
Kopliku et al.Aggregated Search: A New Information RetrievalParadigmACM Computing Surveys, 2014
Ilya Markov [email protected] Information Retrieval 54
![Page 55: Информационный поиск, осень 2016: Федеративный поиск](https://reader034.vdocuments.mx/reader034/viewer/2022051404/587426161a28ab72188b50f1/html5/thumbnails/55.jpg)
Distributed information retrieval Aggregated search Summary
Advanced topics in IR
Data Acquisition
Data Processing
Data Storage
EvaluationRankingQuery Processing
Aggregated Search
Click Models
Present and Future of IR
Offline
Online
Advanced
Ilya Markov [email protected] Information Retrieval 55