1

Term Necessity PredictionP(t | Rq)

Le Zhao and Jamie CallanLanguage Technologies Institute

School of Computer ScienceCarnegie Mellon University

Oct 27, CIKM 2010

• Necessity is as important as idf (theory)

• Explains behavior of IR models (practice)

• Can be predicted

• Performance gain

Main Points

2

Definition of Necessity P(t | Rq)

Directly calculated given relevance judgements for q

Docs that contain t

Relevant (q)

P(t | Rq) = 0.4

Collection

Necessity == 1 – mismatch== term recall

3

Why Necessity?Roots in Probabilistic Models

• Binary Independence Model– [Robertson and Spärck Jones 1976]

– “Relevance Weight”, “Term Relevance”• P(t | R) is effectively the only part about relevance.

Necessity oddsidf (sufficiency)

• Necessity is as important as idf (theory)

• Explains behavior of IR models (practice)

• Can be predicted

• Performance gain

Main Points

Without Necessity

• The emphasis problem for idf-only term weighting– Emphasize high idf terms in query

• “prognosis/viability of a political third party in U.S.” (Topic 206)

4

5

Ground Truth

party political third viability prognosis

True P(t | R) 0.9796 0.7143 0.5918 0.0408 0.0204

idf 2.402 2.513 2.187 5.017 7.471

Emphasis

TREC 4 topic 206

6

Indri Top Results

1. (ZF32-220-147) Recession concerns lead to a discouraging prognosis for 1991

2. (AP880317-0017) Politics … party … Robertson's viability as a candidate

3. (WSJ910703-0174) political parties …

4. (AP880512-0050) there is no viable opposition …

5. (WSJ910815-0072) A third of the votes

6. (WSJ900710-0129) politics, party, two thirds

7. (AP880729-0250) third ranking political movement…

8. (AP881111-0059) political parties

9. (AP880224-0265) prognosis for the Sunday school

10. (ZF32-051-072) third party provider

(Google, Bing still have top 10 false positives. Emphasis also a problem for large search engines!)

7

Without Necessity

• The emphasis problem for idf-only term weighting– Emphasize high idf terms in query

• “prognosis/viability of a political third party in U.S.” (Topic 206)

– False positives throughout rank list• especially detrimental at top rank

– No term recall hurts precision at all recall levels– (This is true for BIM, and also BM25, LM that use tf.)

• How significant is the emphasis problem?

8

Emphasis 64%Mismatch 27%

Precision 9%

Failure Analysis of 44 Topics from TREC 6-8

RIA workshop 2003 (7 top research IR systems, >56 expert*weeks)

Necessity term weighting

Necessity guided expansion

Basis: Term Necessity Prediction

• Necessity is as important as idf (theory)

• Explains behavior of IR models (practice)& Bigrams, &Term restriction using doc fields

• Can be predicted

• Performance gain

Main Points

9

Given True Necessity

• +100% over BIM (in precision at all recall levels)• [Robertson and Spärk Jones 1976]

• +30-80% over Language Model, BM25 (in MAP)• This work

• For a new query w/o relevance judgements, need to predict necessity. – Predictions don’t need to be very accurate to show

performance gain.

10

(Examples from TREC 3 topics)

Term in Query

Oil Spills

Term limitations for US Congress members

Insurance Coverage which pays for Long Term Care

School Choice Voucher System and its effects on the US educational program

Vitamin the cure or cause of human ailments

P(t | R) 0.9914 0.9831 0.6885 0.2821 0.1071

How Necessary are Words?

11

Mismatch Statistics

• Mismatch variation across terms

(TREC 3 title) (TREC 9 desc)

– Not constant, need prediction

stock

com

pute

cost to

y

vouc

hertak

en stop

fund

amen

talism

0

0.2

0.4

0.6

0.8

1Word Necessity

0

0.2

0.4

0.6

0.8

1Word Necessity

12

Mismatch Statistics (2)

• Mismatch variation for the same term in different queries

TREC 3 recurring words

– Query dependent features needed(1/3 term occurrences have necessity variation>0.1)

13

Prior Prediction Approaches

• Croft/Harper combination match (1979)– treats P(t | R) as a tuned constant– when >0.5, rewards docs that match more query terms

• Greiff’s (1998) exploratory data analysis– Used idf to predict overall term weighting– Improved over BIM

• Metzler’s (2008) generalized idf– Used idf to predict P(t | R)– Improved over BIM

• Years of simple idf feature, limited success– Missing piece: P(t | R) = term necessity = term recall

14

Factors that Affect Necessity

What causes a query term to not appear in relevant documents?

• Topic Centrality (Concept Necessity)– E.g., Laser research related or potentially related to US

defense, Welfare laws propounded as reforms

• Synonyms– E.g., movie == film == …

• Abstractness– E.g., Ailments in the vitamin query, Dog Maulings,

Christian Fundamentalism– Worst thing is a rare & abstract term, e.g. prognosis

15

Features

• We need to– Identify synonyms/searchonyms of a query term– in a query dependent way

• Use Thesauri?– Biased (not collection dependent)– Static (not query dependent)– Not promising, Not easy

• Term-term similarity in concept space!– Local LSI (Latent Semantic Indexing)

• LSI of (e.g. 200) top ranked documents• keep (e.g. 150) dimensions

16

Features

• Topic Centrality– Length of term vector after dimension reduction (local

LSI)

• Synonymy (Concept Necessity)– Average similarity scores of top 5 similar terms

• Replaceability– Adjust the Synonymy measure by how many new

documents the synonyms match

• Abstractness– Users modify abstract terms with concrete termseffects on the US educational program prognosis of a political third party

17

Experiments

• Necessity Prediction Error– Regression problem

• Model: RBF kernel regression, M: <f1, f2, .., f5> P(t | R)

• Necessity for Term Weighting– End-to-End retrieval performance– How to weight terms by their necessity

• In BM25– Binary Independence Model

• In Language Models– Relevance model Pm(t | R) – multinomial (Lavrenko and Croft 2001)

18

Necessity Prediction Example

party political third viability prognosis

True P(t | R) 0.9796 0.7143 0.5918 0.0408 0.0204

Predicted 0.7585 0.6523 0.6236 0.3080 0.2869

Emphasis

Trained on TREC 3, tested on TREC 4

19

Necessity Prediction Error

Averag

e (co

nstan

t)

IDF on

ly

All 5 f

eatu

res

Tunin

g meta

-para

meters

TREC 3 rec

urrin

g wor

ds0

0.050.1

0.150.2

0.250.3

0.35

Average Absolute Error (L1 loss) on TREC 4

L1 Loss:

The lower The better• Necessity is as important as idf

• Explains behavior of IR models

• Can be predicted

• Performance gain

Main Points

20

Predicted Necessity Weighting

TREC train sets 3 3-5 3-7 7Test/x-validation 4 6 8 8LM desc – Baseline 0.1789 0.1586 0.1923 0.1923LM desc – Necessity 0.2261 0.1959 0.2314 0.2333Improvement 26.38% 23.52% 20.33% 21.32%

P@10Baseline 0.4160 0.2980 0.3860 0.3860Necessity 0.4940 0.3420 0.4220 0.4380

P@20Baseline 0.3450 0.2440 0.3310 0.3310Necessity 0.4180 0.2900 0.3540 0.3610

10-25% gain (necessity weight)

10-20% gain(top Precision)

21

TREC train sets 3-9 9 11 13Test/x-validation 10 10 12 14LM desc – Baseline 0.1627 0.1627 0.0239 0.1789LM desc – Necessity 0.1813 0.1810 0.0597 0.2233Improvement 11.43% 11.25% 149.8% 24.82%

P@10Baseline 0.3180 0.3180 0.0200 0.4720Necessity 0.3280 0.3400 0.0467 0.5360

P@20Baseline 0.2400 0.2400 0.0211 0.4460Necessity 0.2790 0.2810 0.0411 0.5030

Predicted Necessity Weighting (ctd.)

• Necessity is as important as idf

• Explains behavior of IR models

• Can be predicted

• Performance gain

Main Points

22

vs. Relevance Model

Test/x-validation 4 6 8 8 10 10 12 14

Relevance Model desc 0.2423 0.1799 0.2352 0.2352 0.1888 0.1888 0.0221 0.1774

RM reweight-Only desc 0.2215 0.1705 0.2435 0.2435 0.1700 0.1700 0.0692 0.1945

RM reweight-Trained desc 0.2330 0.1921 0.2542 0.2563 0.1809 0.1793 0.0534 0.2258

Weight Only ≈ ExpansionSupervised > Unsupervised

(5-10%)

Relevance Model: #weight( 1-λ #combine( t1 t2 ) λ #weight( w1 t1

w2 t2

w3 t3

… ) )

x ~ yw1 ~ P(t1|R)w2 ~ P(t2|R)

0 0.2 0.4 0.6 0.8 10

0.20.40.60.8

1

x

y

23

• Necessity is as important as idf (theory)

• Explains behavior of IR models (practice)

• Effective features can predict necessity

• Performance gain

Take Home Messages

24

Acknowledgements• Reviewers from multiple venues• Ni Lao, Frank Lin, Yiming Yang, Stephen Robertson,

Bruce Croft, Matthew Lease– Discussions & references

• David Fisher, Mark Hoy– Maintaining the Lemur toolkit

• Andrea Bastoni and Lorenzo Clemente– Maintaining LSI code for Lemur toolkit

• SVM-light, Stanford parser• TREC

– All the data

• NSF Grant IIS-0707801 and IIS-0534345Feedback: Le Zhao (lezhao@cs.cmu.edu)