modeling and solving term mismatch for full-text retrieval
DESCRIPTION
Modeling and Solving Term Mismatch for Full-Text Retrieval. Dissertation Presentation Le Zhao Language Technologies Institute School of Computer Science Carnegie Mellon University July 26, 2012. Committee:. Jamie Callan (Chair). Jaime Carbonell. Yiming Yang. Bruce Croft (UMass). - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/1.jpg)
Modeling and SolvingTerm Mismatch for Full-Text Retrieval
Dissertation PresentationLe Zhao
Language Technologies InstituteSchool of Computer ScienceCarnegie Mellon University
July 26, 2012
1
Jamie Callan (Chair) Jaime Carbonell Yiming Yang Bruce Croft (UMass)Committee:
![Page 2: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/2.jpg)
2
What is Full-Text Retrieval
• The task
• The Cranfield evaluation [Cleverdon 1960]– abstracts away the user,– allows objective & automatic evaluations
User Query Retrieval Engine
Document Collection
Results User
![Page 3: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/3.jpg)
3
Where are We (Going)?
• Current retrieval models– formal models from 1970s, best ones 1990s– based on simple collection statistics (tf.idf),
no deep understanding of natural language texts• Perfect retrieval
– Query: “information retrieval”, A: “… text search …”
– Textual entailment (difficult natural language task)– Searcher frustration [Feild, Allan and Jones 2010]– Still far away, what have been holding us back?
imply
![Page 4: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/4.jpg)
4
Two Long Standing Problems in Retrieval
• Term mismatch– [Furnas, Landauer, Gomez and Dumais 1987]– No clear definition in retrieval
• Relevance (query dependent term importance – P(t | R))– Traditionally, idf (rareness)– P(t | R) [Robertson and Spärck Jones 1976; Greiff 1998]– Few clues about estimation
• This work– connects the two problems,– shows they can result in huge gains in retrieval,– and uses a predictive approach toward solving both
problems.
![Page 5: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/5.jpg)
5
What is Term Mismatch & Why Care?
• Job search– You look for information retrieval jobs on the market.
They want text search skills.– cost you job opportunities, (50% even if you are careful)
• Legal discovery– You look for bribery or foul play in corporate documents.
They say grease, pay off.– cost you cases
• Patent/Publication search– cost businesses
• Medical record retrieval– cost lives
![Page 6: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/6.jpg)
6
Prior Approaches
• Document:– Full text indexing
• Instead of only indexing key words– Stemming
• Include morphological variants– Document expansion
• Inlink anchor, user tags
• Query:– Query expansion, reformulation
• Both: – Latent Semantic Indexing– Translation based models
![Page 7: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/7.jpg)
7
• Definition
• Significance (theory & practice)
• Mechanism (what causes the problem)
• Model and solution
Main Questions Answered
![Page 8: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/8.jpg)
8
Collection
Definition of Mismatch P(t | Rq)
Directly calculated given relevance judgments for q
Relevant (q)
mismatch (P(t | Rq)) == 1 – term recall (P(t | Rq))_
_
“retrieval”
Jobs mismatched
Documents that contain t
All relevant jobs
Definition Importance Prediction Solution
[CIKM 2010]
![Page 9: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/9.jpg)
9
(Example TREC-3 topics)
Term in Query
Oil Spills
Term limitations for US Congress members
Insurance Coverage which pays for Long Term Care
School Choice Voucher System and its effects on the US educational program
Vitamin the cure or cause of human ailments
P(t | R) 0.9914 0.9831 0.6885 0.2821 0.1071
How Often do Terms Match?
Definition Importance Prediction Solution
![Page 10: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/10.jpg)
10
• Definition• P(t | R) or P(t | R), simple, • estimated from relevant documents, • analyze mismatch
• Significance (theory & practice)
• Mechanism (what causes the problem)
• Model and solution
Main Questions
_
![Page 11: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/11.jpg)
11
Binary Independence Model– [Robertson and Spärck Jones 1976]– Optimal ranking score for each document d
– Term weight for Okapi BM25– Other advanced models behave similarly– Used as effective features in Web search engines
Term Mismatch &Probabilistic Retrieval Models
Idf (rareness)Term recall
Definition Importance: Theory Prediction Solution
![Page 12: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/12.jpg)
12
Binary Independence Model– [Robertson and Spärck Jones 1976]– Optimal ranking score for each document d
– “Relevance Weight”, “Term Relevance”• P(t | R): only part about the query, & relevance
Term Mismatch &Probabilistic Retrieval Models
Definition Importance: Theory Prediction Solution
Term recall Idf (rareness)
![Page 13: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/13.jpg)
13
• Definition
• Significance• Theory (as idf & only part about relevance)• Practice?
• Mechanism (what causes the problem)
• Model and solution
Main Questions
![Page 14: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/14.jpg)
14
Binary Independence Model– [Robertson and Spärck Jones 1976]– Optimal ranking score for each document d
– “Relevance Weight”, “Term Relevance”• P(t | R): only part about the query, & relevance
Term Mismatch &Probabilistic Retrieval Models
Definition Importance: Practice: Mechanism Prediction Solution
Term recall Idf (rareness)
![Page 15: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/15.jpg)
Without Term Recall
• The emphasis problem for tf.idf retrieval models– Emphasize high idf (rare) terms in query
• “prognosis/viability of a political third party in U.S.” (Topic 206)
15
Definition Importance: Practice: Mechanism Prediction Solution
![Page 16: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/16.jpg)
16
Ground Truth (Term Recall)
party political third viability prognosisTrue P(t | R) 0.9796 0.7143 0.5918 0.0408 0.0204
idf 2.402 2.513 2.187 5.017 7.471
Emphasis
Query: prognosis/viability of a political third party
Wrong Emphasis
Definition Importance: Practice: Mechanism Prediction Solution
![Page 17: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/17.jpg)
17
Top Results (Language model)
1. … discouraging prognosis for 1991 …
2. … Politics … party … Robertson's viability as a candidate …
3. … political parties …
4. … there is no viable opposition …
5. … A third of the votes …
6. … politics … party … two thirds …
7. … third ranking political movement…
8. … political parties …
9. … prognosis for the Sunday school …
10. … third party provider …
All are false positives. Emphasis / Mismatch problem, not precision.
( , are better, but still have top 10 false positives. Emphasis / Mismatch also a problem for large search engines!)
Definition Importance: Practice: Mechanism Prediction Solution
Query: prognosis/viability of a political third party
![Page 18: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/18.jpg)
18
Without Term Recall
• The emphasis problem for tf.idf retrieval models– Emphasize high idf (rare) terms in query
• “prognosis/viability of a political third party in U.S.” (Topic 206)
– False positives throughout rank list• especially detrimental at top rank
– No term recall hurts precision at all recall levels
• How significant is the emphasis problem?
Definition Importance: Practice: Mechanism Prediction Solution
![Page 19: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/19.jpg)
19
Emphasis 64%
Precision 9%
Failure Analysis of 44 Topics from TREC 6-8
RIA workshop 2003 (7 top research IR systems, >56 expert*weeks)Failure analyses of retrieval models & techniques still standard today
Recall term weighting
Mismatch guided expansion
Basis: Term Mismatch Prediction
Definition Importance: Practice: Mechanism Prediction Solution
Mismatch 27%
![Page 20: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/20.jpg)
20
• Definition
• Significance• Theory: as idf & only part about relevance• Practice: explains common failures,
other behavior: Personalization, WSD, structured
• Mechanism (what causes the problem)
• Model and solution
Main Questions
![Page 21: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/21.jpg)
21
Emphasis 64%
Precision 9%
Failure Analysis of 44 Topics from TREC 6-8
RIA workshop 2003 (7 top research IR systems, >56 expert*weeks)
Recall term weighting
Mismatch guided expansion
Basis: Term Mismatch Prediction
Definition Importance: Practice: Potential Prediction Solution
Mismatch 27%
![Page 22: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/22.jpg)
22
True Term Recall Effectiveness
• +100% over BIM (in precision at all recall levels)– [Robertson and Spärk Jones 1976]
• +30-80% over Language Model, BM25 (in MAP)– This work
• For a new query w/o relevance judgments, – Need to predict– Predictions don’t need to be very accurate
to show performance gain
Definition Importance: Practice: Potential Prediction Solution
![Page 23: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/23.jpg)
23
• Definition
• Significance• Theory: as idf & only part about relevance• Practice: explains common failures, other behavior,• +30 to 80% potential from term weighting
• Mechanism (what causes the problem)
• Model and solution
Main Questions
![Page 24: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/24.jpg)
24
(Examples from TREC 3 topics)
Term in Query
Oil Spills
Term limitations for US Congress members
Insurance Coverage which pays for Long Term Care
School Choice Voucher System and its effects on the US educational program
Vitamin the cure or cause of human ailments
P(t | R) 0.9914 0.9831 0.6885 0.2821 0.1071
How Often do Terms Match?
idf 5.201 2.010 2.010 1.647 6.405
Differs from idf
Definition Importance Prediction: Idea Solution
Varies 0 to 1
Same term, different Recall
![Page 25: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/25.jpg)
25
Statistics
Term recall across all query terms (average ~55-60%)
TREC 3 titles, 4.9 terms/query TREC 9 descriptions, 6.3 terms/query average 55% term recall average 59% term recall
stock
compu
te cost toy
vouc
hertak
en stop
funda
mental
ism0
0.2
0.4
0.6
0.8
1Term Recall P(t | R)
0
0.2
0.4
0.6
0.8
1Term Recall P(t | R)
Definition Importance Prediction: Idea Solution
![Page 26: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/26.jpg)
26
Statistics
Term recall on shorter queries (average ~70%)
TREC 9 titles, 2.5 terms/query TREC 13 titles, 3.1 terms/query average 70% term recall average 66% term recall
slate
calif
ornia
restri
ct
intell
ig...
freigh
t
pyram
id life
00.10.20.30.40.50.60.70.80.9
1 Term Recall P(t | R)
00.10.20.30.40.50.60.70.80.9
1 Term Recall P(t | R)
Definition Importance Prediction: Idea Solution
![Page 27: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/27.jpg)
27
Query dependent (but for many terms, variance is small)
Statistics
364 recurring words from TREC 3-7, 350 topics
Term Recall for Repeating Terms
Definition Importance Prediction: Idea Solution
![Page 28: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/28.jpg)
P(t | R) vs. idf
28
P(t | R) vs. df/N (Greiff, 1998)
P(t | R)
df/N
-0.5 -0.3 -0.1 0.1 0.3 0.5 0.7 0.9 1.10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1P(t | R)
idf
TREC 4 desc query terms
Definition Importance Prediction: Idea Solution
![Page 29: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/29.jpg)
29
Prior Prediction Approaches
• Croft/Harper combination match (1979)– treats P(t | R) as a tuned constant, or estimated from PRF– when >0.5, rewards docs that match more query terms
• Greiff’s (1998) exploratory data analysis– Used idf to predict overall term weighting– Improved over basic BIM
• Metzler’s (2008) generalized idf– Used idf to predict P(t | R)– Improved over basic BIM
• Simple feature (idf), limited success– Missing piece: P(t | R) = term recall = 1 – term mismatch
Definition Importance Prediction: Idea Solution
![Page 30: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/30.jpg)
30
What Factors can Cause Mismatch?
• Topic centrality (Is concept central to topic?)– “Laser research related or potentially related to defense”– “Welfare laws propounded as reforms”
• Synonyms (How often they replace original term?)– “retrieval” == “search” == …
• Abstractness– “Laser research … defense”
“Welfare laws”– “Prognosis/viability” (rare & abstract)
Definition Importance Prediction: Idea Solution
![Page 31: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/31.jpg)
31
• Definition
• Significance
• Mechanism• Causes of mismatch: Unnecessary concepts,
replaced by synonyms or more specific terms
• Model and solution
Main Questions
![Page 32: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/32.jpg)
32
Designing Features to Model the Factors
• We need to– Identify synonyms/searchonyms of a query term– in a query dependent way
• External resource? (WordNet, wiki, or query log)– Biased (coverage problem, collection independent)– Static (not query dependent)– Not easy, not used here
• Term-term similarity in concept space!– Local LSI (Latent Semantic Indexing)
Definition Importance Prediction: Implement Solution
Query Retrieval Engine
Document Collection
ResultsResults
Top (500) Results
Concept Space
(150 dim)
![Page 33: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/33.jpg)
33
term limit
ballot
elect te
rm longnurse car
e ail
health
disease
basler
0
0.1
0.2
0.3
0.4
0.5
Top Similar Terms
Similarity with query term
Synonyms from Local LSI
Term limitation for US Congress members
Insurance Coverage which pays for Long Term Care
Vitamin the cure or cause of human ailments
0.9831 0.6885 0.1071P(t | Rq)
Definition Importance Prediction: Implement Solution
![Page 34: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/34.jpg)
34
term limit
ballot
elect te
rm longnurse car
e ail
health
disease
basler
0
0.1
0.2
0.3
0.4
0.5
Top Similar Terms
Similarity with query term
Synonyms from Local LSI
Term limitation for US Congress members
Insurance Coverage which pays for Long Term Care
Vitamin the cure or cause of human ailments
0.9831 0.6885 0.1071
(1) Magnitude of self similarity – Term centrality
(2) Avg similarity of supporting terms – Concept centrality
(3) How likely synonyms replace term t in collection
P(t | Rq)
Definition Importance Prediction: Implement Solution
![Page 35: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/35.jpg)
35
Features that Model the Factors
• Term centrality– Self-similarity (length of t) after dimension reduction
• Concept centrality– Avg similarity of supporting terms (top synonyms)
• Replaceability– How frequently synonyms appear in place of original
query term in collection documents• Abstractness
– Users modify abstract terms with concrete termseffects on the US educational program prognosis of a political third party
Correlation with P(t | R)0.3719
0.3758
– 0.1872
– 0.1278
Definition Importance Prediction: Experiment Solution
idf: – 0.1339
![Page 36: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/36.jpg)
36
Prediction Model
Regression modeling– Model:
M: <f1, f2, .., f5> P(t | R)– Train on one set of queries (known relevance), – Test on another set of queries (unknown relevance)– RBF kernel Support Vector regression
Definition Importance Prediction: Implement Solution
![Page 37: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/37.jpg)
37
A General View of Retrieval Modeling as Transfer Learning
• The traditional restricted view sees a retrieval model as– a document classifier for a given query.
• More general view: A retrieval model really is– a meta-classifier, responsible for many queries,– mapping a query to a document classifier.
• Learning a retrieval model == transfer learning– Using knowledge from related tasks (training queries)
to classify documents for a new task (test query)– Our features and model facilitate the transfer.– More general view more principled investigations
and more advanced techniques
Definition Importance Prediction Solution
![Page 38: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/38.jpg)
38
Experiments
• Term recall prediction error– L1 loss (absolute prediction error)
• Term recall based term weighting retrieval – Mean Average Precision (overall retrieval success)– Precision at top 10 (precision at top of rank list)
Definition Importance Prediction: Experiment Solution
![Page 39: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/39.jpg)
39
Term Recall Prediction Example
party political third viability prognosisTrue P(t | R) 0.9796 0.7143 0.5918 0.0408 0.0204
Predicted 0.7585 0.6523 0.6236 0.3080 0.2869
Emphasis
Query: prognosis/viability of a political third party.(Trained on TREC 3)
Definition Importance Prediction: Experiment Solution
![Page 40: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/40.jpg)
40
Term Recall Prediction Error
Averag
e (co
nstan
t)
IDF on
ly
All 5 fe
atures
Tuning
meta
-param
eters
TREC 3 rec
urring
word
s0
0.1
0.2
0.3
Average Absolute Error (L1 loss) on TREC 4
L1 Loss:
The lower, the better
Definition Importance Prediction: Experiment Solution
![Page 41: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/41.jpg)
41
• Definition
• Significance
• Mechanism
• Model and solution• Can be predicted;
Framework to design and evaluate features
Main Questions
![Page 42: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/42.jpg)
42
Using (t | R) in Retrieval Models
• In BM25– Binary Independence Model
• In Language Modeling (LM)– Relevance Model [Lavrenko and Croft 2001]
Definition Importance Prediction Solution: Weighting
Only term weighting, no expansion.
![Page 43: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/43.jpg)
43
Predicted Recall Weighting10-25% gain
(MAP)
Definition Importance Prediction Solution: Weighting
“*”: significantly better by sign & randomization tests
Datasets: train -> test
3 -> 4 3-5 -> 6 3-7 -> 8 7 -> 8 3-9 -> 10 9 -> 10 11 -> 12 13 -> 140
0.05
0.1
0.15
0.2
0.25Baseline LM descNecessity LM desc
MAP
**
*
*
**
*Recall LM desc
![Page 44: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/44.jpg)
44
Predicted Recall Weighting10-20% gain
(top Precision)
Definition Importance Prediction Solution: Weighting
3 -> 4 3-5 -> 6 3-7 -> 8 7 -> 8 3-9 -> 10 9 -> 10 11 -> 12 13 -> 140
0.1
0.2
0.3
0.4
0.5
0.6Baseline LM descNecessity LM desc
Prec@10
*
*
!!!Recall LM desc
“*”: Prec@10 is significantly better.“!”: Prec@20 is significantly better.
Datasets: train -> test
![Page 45: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/45.jpg)
45
0 0.2 0.4 0.6 0.8 10
0.20.40.60.8
1
PredictedTerm Recall
Relevance Model Weights (normalized)
TREC 3
vs. Relevance Model
Relevance Model [Laverenko and Croft 2001]
x
yRM weight (x) ~ Term recall (y)
Definition Importance Prediction Solution: Weighting
0 0.2 0.4 0.6 0.8 10
0.20.40.60.8
1
PredictedTerm Recall
Relevance Model Weights (normalized)
TREC 3TREC 7
TREC 13
Query LikelihoodUnsupervisedTerm occurrence in top docs
5-10% better than unsupervised
Pm(t1 | R) Pm(t2 | R)
~ P(t1 | R)~ P(t2 | R)
![Page 46: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/46.jpg)
46
• Definition
• Significance
• Mechanism
• Model and solution• Term weighting solves emphasis problem for long
queries• Mismatch problem?
Main Questions
![Page 47: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/47.jpg)
47
Emphasis 64%
Precision 9%
Failure Analysis of 44 Topics from TREC 6-8
RIA workshop 2003 (7 top research IR systems, >56 expert*weeks)
Recall term weighting
Mismatch guided expansion
Basis: Term Mismatch Prediction
Definition Importance Prediction Solution: Expansion
Mismatch 27%
![Page 48: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/48.jpg)
48
Recap: Term Mismatch
• Term mismatch ranges 30%-50% on average• Relevance matching can degrade quickly for multi-word
queries
• Solution: Fix every query term
Definition Importance Prediction Solution: Expansion
[SIGIR 2012]
![Page 49: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/49.jpg)
49
Conjunctive Normal Form (CNF) Expansion
Example keyword query:placement of cigarette signs on television watched by children
Manual CNF: (placement OR place OR promotion OR logo OR sign OR signage OR merchandise)AND (cigarette OR cigar OR tobacco)AND (television OR TV OR cable OR network)AND (watch OR view)AND (children OR teen OR juvenile OR kid OR adolescent)
– Expressive & compact (1 CNF == 100s alternatives)– Highly effective (this work: 50-300% over base keyword)– Used by lawyers, librarians and other expert searchers– But, tedious & difficult to create, little research
Definition Importance Prediction Solution: Expansion
![Page 50: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/50.jpg)
Diagnostic Intervention
• Goal– Least amount user effort near optimal performance– E.g. expand 2 terms 90% of total improvement
50
Definition Importance Prediction Solution: Expansion
placement of cigarette signs on television watched by children
placement of cigarette signs on television watched by children
High idf (rare) terms
CNF (placement OR place OR promotion OR sign OR signage OR merchandise)
AND cigar AND television AND watchAND (children OR teen OR juvenile OR kid OR adolescent)
(placement OR place OR promotion OR sign OR signage OR merchandise)
AND cigar AND (television OR tv OR cable OR network)AND watch AND children
Query: placement of cigarette signs on television watched by children
Diagnosis:
Expansion:
Low terms
CNF
![Page 51: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/51.jpg)
Diagnostic Intervention
• Goal– Least amount user effort near optimal performance– E.g. expand 2 terms 90% of total improvement
51
Definition Importance Prediction Solution: Expansion
placement of cigarette signs on television watched by children
placement of cigarette signs on television watched by children
[ 0.9 (placement cigar television watch children) 0.1 (0.4 place 0.3 promotion 0.2 logo 0.1 sign 0.3 signage 0.3 merchandise 0.5 teen 0.4 juvenile 0.2 kid 0.1 adolescent) ]
[ 0.9 (placement cigar television watch children) 0.1 (0.4 place 0.3 promotion 0.2 logo 0.1 sign 0.3 signage 0.3 merchandise 0.5 tv 0.4 cable 0.2 network) ]
Query: placement of cigarette signs on television watched by children
Diagnosis:
Expansion query
Bag of wordExpansion: Bag of wordOriginal query
High idf (rare) termsLow terms
![Page 52: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/52.jpg)
52
Diagnostic Intervention (We Hope to)
User Keyword queryDiagnosis
system(P(t | R) or idf)
Problem query termsUser expansionExpansion
terms
Query formulation
(CNF or Keyword)
Retrieval engine Evaluation
(child AND cigar)
(child > cigar)
(child OR teen) AND cigar
(child teen)
Definition Importance Prediction Solution: Expansion
![Page 53: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/53.jpg)
53
Diagnostic Intervention (We Hope to)
User Keyword queryDiagnosis
system(P(t | R) or idf)
Problem query termsUser expansionExpansion
terms
Query formulation
(CNF or Keyword)
Retrieval engine Evaluation
(child AND cigar)
(child > cigar)
(child OR teen) AND cigar
(child teen)
Definition Importance Prediction Solution: Expansion
![Page 54: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/54.jpg)
54
Expert user Keyword queryDiagnosis
system(P(t | R) or idf)
Problem query termsUser expansionExpansion
terms
Query formulation
(CNF or Keyword)
Retrieval engine Evaluation
Online simulation
Online simulation
We Ended up Using Simulation
(child teen)
(child OR teen) AND cigar
(child OR teen) AND (cigar OR tobacco)
FullCNFOffline
(child AND cigar)
(child > cigar)
Definition Importance Prediction Solution: Expansion
![Page 55: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/55.jpg)
55
Diagnostic Intervention Datasets
• Document sets– TREC 2007 Legal track, 7 million tobacco company– TREC 4 Ad hoc track, 0.5 million newswire
• CNF Queries, 50 topics per dataset– TREC 2007 by lawyers, TREC 4 by Univ. Waterloo
• Relevance Judgments– TREC 2007 sparse, TREC 4 dense
• Evaluation measures– TREC 2007 statAP, TREC 4 MAP
Definition Importance Prediction Solution: Expansion
![Page 56: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/56.jpg)
56
P(t | R) vs. idf diagnosis
Results – Diagnosis
Diagnostic CNF expansion on TREC 4 and 2007
0 1 2 3 4 All0%
10%20%30%40%50%60%70%80%90%
100%
P(t | R) on TREC 2007idf on TREC 2007P(t | R) on TREC 4idf on TREC 4
# query terms selected
Gain in retrieval (MAP)
Definition Importance Prediction Solution: Expansion
No Expansion
Full Expansion
![Page 57: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/57.jpg)
57
Results – Form of Expansion
CNF vs. bag-of-word expansion
0 1 2 3 4 All0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
CNF on TREC 4Bag of word on TREC 4CNF on TREC 2007Bag of word on TREC 2007
# query terms selected
Retrieval performance (MAP)
P(t | R) guided expansion on TREC 4 and 2007
Definition Importance Prediction Solution: Expansion
50% to300%gain
Similar level of gain in top precision
![Page 58: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/58.jpg)
58
• Definition
• Significance
• Mechanism
• Model and solution• Term weighting for long queries• Term mismatch prediction diagnoses problem terms,
and produces simple & effective CNF queries
Main Questions
![Page 59: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/59.jpg)
59
Efficient P(t | R) Prediction
• 3-10X speedup (close to simple keyword retrieval), while maintaining 70-90% of the gain
• Predict using P(t | R) values from similar, previously-seen queries
[CIKM 2012]
Definition Importance Prediction: Efficiency Solution: Weighting
![Page 60: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/60.jpg)
60
Contributions
• Two long standing problems: mismatch & P(t | R)• Definition and initial quantitative analysis of mismatch
– Do better/new features and prediction methods exist?• Role of term mismatch in basic retrieval theory
– Principled ways to solve term mismatch– What about advanced learning to rank, transfer learning?
• Ways to automatically predict term mismatch– Initial modeling of causes of mismatch, features– Efficient prediction using historic information– Are there better analyses or modeling of the causes?
Definition Importance Prediction Solution
![Page 61: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/61.jpg)
61
Contributions
• Effectiveness of ad hoc retrieval– Term weighting & diagnostic expansion– How to do automatic CNF expansion?– Better formalisms: transfer learning, & more tasks?
• Diagnostic intervention– Mismatch diagnosis guides targeted expansion– How to diagnose specific types of mismatch problems
or different problems (mismatch/emphasis/precision)?• Guide NLP, Personalization, etc. to solve the real problem
– How to proactively identify search and other user needs?
Definition Importance Prediction Solution
![Page 62: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/62.jpg)
62
Acknowledgements• Committee: Jamie Callan, Jaime Carbonell, Yiming Yang, Bruce Croft• Ni Lao, Frank Lin, Siddharth Gopal, Jon Elsas, Jaime Arguello, Hui (Grace)
Yang, Stephen Robertson, Matthew Lease, Nick Craswell, Yi Zhang (and her group), Jin Young Kim, Yangbo Zhu, Runting Shi, Yi Wu, Hui Tan, Yifan Yanggong, Mingyan Fan, Chengtao Wen
– Discussions & references & feedback
• Reviewers: papers & NSF proposal• David Fisher, Mark Hoy, David Pane
– Maintaining the Lemur toolkit
• Andrea Bastoni and Lorenzo Clemente– Maintaining LSI code for Lemur toolkit
• SVM-light, Stanford parser• TREC: data• NSF Grant IIS-1018317• Xiangmin Jin, and my whole family
and volleyball packs at CMU & SF Bay
![Page 63: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/63.jpg)
63
END
![Page 64: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/64.jpg)
64
Prior Definition of Mismatch
• Vocabulary mismatch (Furnas et al., 1987)– How likely 2 people disagree in vocab choice– Domain experts disagree 80-90% of the times– Leads to Latent Semantic Indexing (Deerwester et al.,
1988)– Query independent– = Avgq P(t | Rq)– can be reduced to our query dependent definition of
term mismatch-
![Page 65: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/65.jpg)
65
KnowledgeHow Necessity explains behavior of IR techniques
• Why weight query bigrams 0.1, while query unigrams 0.9?– Bigram decreases term recall, weight reflects recall
• Why Bigram not gaining stable improvements?– Term recall is more of a problem
• Why using document structure (field, semantic annotation) not improving performance?– Improves precision, need to solve structural mismatch
• Word sense disambiguation– Enhances precision, instead, should use in mismatch modeling!
• Identify query term sense, for searchonym id, or learning across queries• Disambiguate collection term sense for more accurate replaceability
• Personalization– biases results to what a community/person likes to read (precision)– may work well in a mobile setting, short queries
![Page 66: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/66.jpg)
66
Why Necessity?System Failure Analysis
• Reliable Information Access (RIA) workshop (2003)– Failure analysis for 7 top research IR systems
• 11 groups of researchers (both academia & industry)• 28 people directly involved in the analysis (senior & junior)• >56 human*weeks (analysis + running experiments)• 45 topics selected from 150 TREC 6-8 (difficult topics)
– Causes (necessity in various disguise)• Emphasize 1 aspect, missing another aspect (14+2 topics)• Emphasize 1 aspect, missing another term (7 topics)• Missing either 1 of 2 aspects, need both (5 topics)• Missing difficult aspect that need human help(7 topics)• Need to expand a general term e.g. “Europe” (4 topics)• Precision problem, e.g. “euro”, not “euro-…” (4 topics)
![Page 67: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/67.jpg)
67
![Page 68: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/68.jpg)
68
![Page 69: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/69.jpg)
69
Local LSI Top Similar Terms
Oil spills Insurance coverage which pays for long term care
Term limitations for US Congress members
Vitamin the cure of or cause for human ailments
oil term term ailspill 0.5828 term 0.3310 term 0.3339 ail 0.4415
oil 0.4210 long 0.2173 limit 0.1696 health 0.0825
tank 0.0986 nurse 0.2114 ballot 0.1115 disease 0.0720
crude 0.0972 care 0.1694 elect 0.1042 basler 0.0718
water 0.0830 home 0.1268 care 0.0997 dr 0.0695
![Page 70: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/70.jpg)
70
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
1.2 Error plot of necessity predictions
Necessity truthPredicted necessityPrediction trend (3rd order polynomial fit)
Prob
abili
ty
![Page 71: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/71.jpg)
71
Necessity vs. idf (and emphasis)
![Page 72: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/72.jpg)
72
True Necessity Weighting
TREC 4 6 8 9 10 12 14Document collection disk 2,3 disk 4,5 d4,5 w/o cr WT10g .GOV .GOV2
Topic numbers 201-250 301-350 401-450 451-500 501-550 TD1-50 751-800
LM desc – Baseline 0.1789 0.1586 0.1923 0.2145 0.1627 0.0239 0.1789
LM desc – Necessity 0.2703 0.2808 0.3057 0.2770 0.2216 0.0868 0.2674Improvement 51.09% 77.05% 58.97% 29.14% 36.20% 261.7% 49.47%p - randomization 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0001
p - sign test 0.0000 0.0000 0.0000 0.0005 0.0000 0.0000 0.0002
Multinomial-abs 0.1988 0.2088 0.2345 0.2239 0.1653 0.0645 0.2150Multinomial RM 0.2613 0.2660 0.2969 0.2590 0.2259 0.1219 0.2260
Okapi desc – Baseline 0.2055 0.1773 0.2183 0.1944 0.1591 0.0449 0.2058
Okapi desc – Necessity 0.2679 0.2786 0.2894 0.2387 0.2003 0.0776 0.2403LM title – Baseline N/A 0.2362 0.2518 0.1890 0.1577 0.0964 0.2511
LM title – Necessity N/A 0.2514 0.2606 0.2058 0.2137 0.1042 0.2674
![Page 73: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/73.jpg)
73
Predicted Necessity Weighting10-25% gain
(necessity weight)10-20% gain
(top Precision)
TREC train sets 3 3-5 3-7 7Test/x-validation 4 6 8 8LM desc – Baseline 0.1789 0.1586 0.1923 0.1923LM desc – Necessity 0.2261 0.1959 0.2314 0.2333Improvement 26.38% 23.52% 20.33% 21.32%
P@10 Baseline 0.4160 0.2980 0.3860 0.3860Necessity 0.4940 0.3420 0.4220 0.4380
P@20 Baseline 0.3450 0.2440 0.3310 0.3310Necessity 0.4180 0.2900 0.3540 0.3610
![Page 74: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/74.jpg)
74
TREC train sets 3-9 9 11 13Test/x-validation 10 10 12 14LM desc – Baseline 0.1627 0.1627 0.0239 0.1789LM desc – Necessity 0.1813 0.1810 0.0597 0.2233Improvement 11.43% 11.25% 149.8% 24.82%
P@10 Baseline 0.3180 0.3180 0.0200 0.4720Necessity 0.3280 0.3400 0.0467 0.5360
P@20 Baseline 0.2400 0.2400 0.0211 0.4460Necessity 0.2790 0.2810 0.0411 0.5030
Predicted Necessity Weighting (ctd.)
![Page 75: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/75.jpg)
75
vs. Relevance Model
Weight Only ≈ ExpansionSupervised > Unsupervised
(5-10%)
Relevance Model: #weight( 1-λ #combine( t1 t2 ) λ #weight( w1 t1
w2 t2
w3 t3
… ) )
x ~ yw1 ~ P(t1|R)w2 ~ P(t2|R)
0 0.2 0.4 0.6 0.8 10
0.20.40.60.8
1
x
y
Test/x-validation 4 6 8 8 10 10 12 14LM desc – Baseline 0.1789 0.1586 0.1923 0.1923 0.1627 0.1627 0.0239 0.1789Relevance Model desc 0.2423 0.1799 0.2352 0.2352 0.1888 0.1888 0.0221 0.1774RM reweight-Only desc 0.2215 0.1705 0.2435 0.2435 0.1700 0.1700 0.0692 0.1945RM reweight-Trained desc 0.2330 0.1921 0.2542 0.2563 0.1809 0.1793 0.0534 0.2258
![Page 76: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/76.jpg)
76
Feature Correlation
f1 Term f2 Con f3 Repl f4 DepLeaf f5 idf RMw
0.3719 0.3758 -0.1872 0.1278 -0.1339 0.6296
Predicted Necessity: 0.7989 (TREC 4 test set)
≈
![Page 77: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/77.jpg)
77
3 -> 4 3-5 -> 6 3-7 -> 8 7 -> 8 3-9 -> 10 9 -> 10 11 -> 12 13 -> 140
0.05
0.1
0.15
0.2
0.25
0.3
RM Reweight-Trained desc3 -> 4 3-5 -> 6 3-7 -> 8 7 -> 8 3-9 -> 10 9 -> 10 11 -> 12 13 -> 14
0
0.05
0.1
0.15
0.2
0.25
0.3
Baseline LM descRelevance Model desc
MAPMAP
vs. Relevance Model
3 -> 4 3-5 -> 6 3-7 -> 8 7 -> 8 3-9 -> 10 9 -> 10 11 -> 12 13 -> 140
0.05
0.1
0.15
0.2
0.25
0.3
RM Reweight-Only desc
Weight Only ≈ Expansion
RM is unstable
Supervised > Unsupervised
(5-10%)
Datasets: train -> test
![Page 78: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/78.jpg)
78
Efficient Prediction of Term Recall
• Currently:– slow query dependent features that requires retrieval
• Can they be more effective and more efficient?– Need to understand the causes of the query
dependent variation– Design a minimal set of efficient features to capture
the query dependent variations
Definition Importance Prediction: Idea Solution
![Page 79: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/79.jpg)
79
Causes of Query Dependent Variation (1)
• Example
• Cause– Different word sense
![Page 80: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/80.jpg)
80
Causes of Query Dependent Variation (2)
• Example
• Cause– Different word use, e.g. term in phrase vs. not
![Page 81: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/81.jpg)
81
Causes of Query Dependent Variation (3)
• Example
• Cause– Different Boolean semantics of the queries, AND vs.
OR
![Page 82: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/82.jpg)
82
Causes of Query Dependent Variation (4)
• Example
• Cause– Different association level with topic
![Page 83: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/83.jpg)
83
Efficient P(t | R) Prediction (2)
• Causes of P(t | R) variation of same term in different queries– Different query semantics: Canada or Mexico vs.
Canada– Different word sense: bear (verb) vs. bear (noun)– Different word use: Seasonal affective disorder
syndrome (SADS) vs. Agoraphobia as a disorder– Difference in association level with topic
• Use historic occurrences to predict current– 70-90% of the total gain– 3-10X faster, close to simple keyword retrieval
![Page 84: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/84.jpg)
84
Efficient P(t | R) Prediction (2)
• Low variation of same term in different queries• Use historic occurrences to predict current
– 3-10X faster, close to the slower method & real time
3 -> 4 3-5 -> 6 3-7 -> 8 7 -> 8 3-9 -> 10
9 -> 10 11 -> 12 13 -> 140
0.05
0.1
0.15
0.2
0.25
0.3 Baseline LM descNecessity LM descEfficient Prediction
MAP
*
*
**
train -> test
![Page 85: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/85.jpg)
85
Using Document Structure
• Stylistic: XML• Syntactic/Semantic: POS, Semantic Role Label• Current approaches
– All precision oriented• Need to solve mismatch first?
![Page 86: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/86.jpg)
86Collections
Retrieval Model
Motivation
• Search is important, information portal• Search is research worthy
– SIGIR, WWW, CIKM, ASIST, ECIR, AIRS, – Search is difficult
• Retrieval modeling difficulty >= sentence paraphrasing– Since 1970s, but still not fully understood, basic problem like mismatch– Adapt to changing requirements of mobile, social and semantic Web
• Modeling user’s needsUser
QueryResultsActivities
User Query Retrieval Model
Document Collection
Results
![Page 87: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/87.jpg)
87
Online or Offline Study?
• Controlling confounding variables– Quality of expansion terms– User’s prior knowledge of the topic– Interaction form & effort
• Enrolling many users and repeating experiments• Offline simulations can avoid all these and still make
reasonable observations
![Page 88: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/88.jpg)
88
Simulation Assumptions
• Real full CNFs to simulate partial expansions• 3 assumptions about user expansion process
– Expansion of individual terms are independent of each other• A1: always same set of expansion terms for a given query
term, no matter which subset of query terms get expanded.• A2: same sequence of expansion terms, no matter …
– A3: Keyword query is re-constructed from the CNF query• Procedure to ensure vocabulary faithful to that of the original
keyword description• Highly effective CNF queries ensure reasonable kw baseline
![Page 89: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/89.jpg)
89
Take Home Message for Ordinary Search Users (people and software)
![Page 90: Modeling and Solving Term Mismatch for Full-Text Retrieval](https://reader035.vdocuments.mx/reader035/viewer/2022081604/56816857550346895dde7f56/html5/thumbnails/90.jpg)
90
Be mean! Is the term Necessary for
doc relevance?
If not, remove, replace or expand.