![Page 1: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/1.jpg)
3
Search Engines
Provide a ranked list of documents.
May provide relevance scores.
May have performance information.
![Page 2: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/2.jpg)
7
External Metasearch
Search Engine
A
Metasearch Engine
Search Engine
B
Search Engine
C
Database A
Database C
Database B
![Page 3: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/3.jpg)
8
Internal Metasearch
Text Module
Metasearch core
URL Module
Image Module
HTML Database
Image Database
Search Engine
![Page 4: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/4.jpg)
9
Metasearch Engines
Query multiple search engines.
May or may not combine results.
![Page 5: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/5.jpg)
13
Outline
Introduce problem
Characterize problem
Survey techniques
Upper bounds for metasearch
![Page 6: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/6.jpg)
14
Characterizing Metasearch
Three axes:
common vs. disjoint database,
relevance scores vs. ranks,
training data vs. no training data.
![Page 7: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/7.jpg)
15
Axis 1: DB Overlap
High overlap
data fusion.
Low overlap
collection fusion (distributed retrieval).
Very different techniques for each…
Today: data fusion.
![Page 8: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/8.jpg)
16
Classes of Metasearch Problems
no training
data
training
data
rele
vance
score
s
ranks
only
CombMNZ LC model
Bayes Borda,
Condorcet,
rCombMNZ
![Page 9: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/9.jpg)
17
Outline
Introduce problem
Characterize problem
Survey techniques
Upper bounds for metasearch
![Page 10: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/10.jpg)
18
Classes of Metasearch Problems
no training
data
training
data
rele
vance
score
s
ranks
only
CombMNZ LC model
Bayes Borda,
Condorcet,
rCombMNZ
![Page 11: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/11.jpg)
19
CombSUM [Fox, Shaw, Lee, et al.]
Normalize scores: [0,1].
For each doc:
sum relevance scores given to it by each
system (use 0 if unretrieved).
Rank documents by score.
Variants: MIN, MAX, MED, ANZ, MNZ
![Page 12: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/12.jpg)
20
CombMNZ [Fox, Shaw, Lee, et al.]
Normalize scores: [0,1].
For each doc:
sum relevance scores given to it by each
system (use 0 if unretrieved), and
multiply by number of systems that
retrieved it (MNZ).
Rank documents by score.
![Page 13: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/13.jpg)
21
How well do they perform?
Need performance metric.
Need benchmark data.
![Page 14: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/14.jpg)
22
Metric: Average Precision
R
N
R
N
R
N
1/1
2/3
3/5
4/8
0.6917
![Page 15: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/15.jpg)
23
Benchmark Data: TREC
Annual Text Retrieval Conference.
Millions of documents (AP, NYT, etc.)
50 queries.
Dozens of retrieval engines.
Output lists available.
Relevance judgments available.
![Page 16: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/16.jpg)
24
Data Sets
Data set Number
systems
Number
queries
Number of
docs
TREC3 40 50 1000
TREC5 61 50 1000
Vogt 10 10 1000
TREC9 105 50 1000
![Page 17: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/17.jpg)
25
CombX on TREC5 Data
![Page 18: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/18.jpg)
26
CombX on TREC5 Data, II
![Page 19: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/19.jpg)
27
Experiments
Randomly choose n input systems.
For each query:
combine, trim, calculate avg precision.
Calculate mean avg precision.
Note best input system.
Repeat (statistical significance).
![Page 20: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/20.jpg)
28
CombMNZ on TREC3
![Page 21: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/21.jpg)
29
CombMNZ on TREC5
![Page 22: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/22.jpg)
30
CombMNZ on Vogt
![Page 23: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/23.jpg)
31
CombMNZ on TREC9
![Page 24: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/24.jpg)
32
Metasearch via Voting [Aslam, Montague]
Analog to election strategies.
Requires only rank information.
No training required.
![Page 25: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/25.jpg)
33
Classes of Metasearch Problems
no training
data
training
data
rele
vance
score
s
ranks
only
CombMNZ LC model
Bayes Borda,
Condorcet,
rCombMNZ
![Page 26: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/26.jpg)
34
Election Strategies
Plurality vote.
Approval vote.
Run-off.
Preferential rankings:
instant run-off,
Borda count (positional),
Condorcet method (head-to-head).
![Page 27: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/27.jpg)
35
Metasearch Analogy
Documents are candidates.
Systems are voters expressing preferential rankings among candidates.
![Page 28: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/28.jpg)
36
Borda Count
Consider an n candidate election.
One method for choosing winner is the Borda count. [Borda, Saari]
For each voter i
Assign n points to top candidate.
Assign n-1 points to next candidate.
…
Rank candidates according to point sum.
![Page 29: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/29.jpg)
37
Election 2000: Florida
![Page 30: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/30.jpg)
38
Borda Count: Election 2000
Ideological order: Nader, Gore, Bush.
Ideological voting:
Bush voter: Bush, Gore, Nader.
Nader voter: Nader, Gore, Bush.
Gore voter:
Gore, Bush, Nader.
Gore, Nader, Bush. 50/50, 100/0
![Page 31: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/31.jpg)
39
Election 2000: Ideological Florida Voting
Gore Bush Nader
50/50 14,734,379 13,185,542 7,560,864
100/0 14,734,379 14,639,267 6,107,138
Gore Wins
![Page 32: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/32.jpg)
40
Borda Count: Election 2000
Ideological order: Nader, Gore, Bush.
Manipulative voting:
Bush voter: Bush, Nader, Gore.
Gore voter: Gore, Nader, Bush.
Nader voter: Nader, Gore, Bush.
![Page 33: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/33.jpg)
41
Election 2000: Manipulative Florida Voting
Gore Bush Nader
11,825,203 11,731,816 11,923,765
Nader Wins
![Page 34: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/34.jpg)
42
Metasearch via Borda Counts
Metasearch analogy:
Documents are candidates.
Systems are voters providing preferential rankings.
Issues:
Systems may rank different document sets.
How to deal with unranked documents?
![Page 35: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/35.jpg)
43
Borda on TREC5 Data, I
![Page 36: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/36.jpg)
44
Borda on TREC5 Data, II
![Page 37: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/37.jpg)
45
Borda on TREC5 Data, III
![Page 38: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/38.jpg)
46
Condorcet Voting
Each ballot ranks all candidates.
Simulate head-to-head run-off between each pair of candidates.
Condorcet winner: candidate that beats all other candidates, head-to-head.
![Page 39: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/39.jpg)
47
Election 2000: Florida
![Page 40: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/40.jpg)
48
Condorcet Paradox
Voter 1: A, B, C
Voter 2: B, C, A
Voter 3: C, A, B
Cyclic preferences: cycle in Condorcet graph.
Condorcet consistent path: Hamiltonian.
For metasearch: any CC path will do.
![Page 41: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/41.jpg)
49
Condorcet Consistent Path
![Page 42: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/42.jpg)
50
Hamiltonian Path Proof
Base Case:
Inductive Step:
![Page 43: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/43.jpg)
51
Condorcet-fuse: Sorting
Insertion-sort suggested by proof.
Quicksort too; O(n log n) comparisons.
n documents.
Each comparison: O(m).
m input systems.
Total: O(m n log n).
Need not compute entire graph.
![Page 44: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/44.jpg)
52
Condorcet-fuse on TREC3
![Page 45: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/45.jpg)
53
Condorcet-fuse on TREC5
![Page 46: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/46.jpg)
54
Condorcet-fuse on Vogt
![Page 47: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/47.jpg)
55
Condorcet-fuse on TREC9
![Page 48: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/48.jpg)
56
Outline
Introduce problem
Characterize problem
Survey techniques
Upper bounds for metasearch
![Page 49: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/49.jpg)
57
Upper Bounds on Metasearch
How good can metasearch be?
Are there fundamental limits that methods are approaching?
![Page 50: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/50.jpg)
58
Upper Bounds on Metasearch
Constrained oracle model:
omniscient metasearch oracle,
constraints placed on oracle that any reasonable metasearch technique must
obey.
What are “reasonable” constraints?
![Page 51: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/51.jpg)
59
Naïve Constraint
Naïve constraint:
Oracle may only return docs from
underlying lists.
Oracle may return these docs in any order.
Omniscient oracle will return relevant docs above irrelevant docs.
![Page 52: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/52.jpg)
60
TREC5: Naïve Bound
![Page 53: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/53.jpg)
61
Pareto Constraint
Pareto constraint:
Oracle may only return docs from
underlying lists.
Oracle must respect unanimous will of
underlying systems.
Omniscient oracle will return relevant docs
above irrelevant docs, subject to the above constraint.
![Page 54: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/54.jpg)
62
TREC5: Pareto Bound
![Page 55: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/55.jpg)
63
Majoritarian Constraint
Majoritarian constraint:
Oracle may only return docs from
underlying lists.
Oracle must respect majority will of
underlying systems.
Omniscient oracle will return relevant docs
above irrelevant docs and break cycles optimally, subject to the above constraint.
![Page 56: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/56.jpg)
64
TREC5: Majoritarian Bound
![Page 57: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/57.jpg)
65
Upper Bounds: TREC3
![Page 58: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/58.jpg)
66
Upper Bounds: Vogt
![Page 59: Search Engines - Northeastern University · Search Engines Provide a ... Dozens of retrieval engines. Output lists available. Relevance judgments available. 24 Data Sets Data set](https://reader034.vdocuments.mx/reader034/viewer/2022051310/6007164c6bdce628de4bf00b/html5/thumbnails/59.jpg)
67
Upper Bounds: TREC9