axiomatic analysis of smoothing methods in language models for pseudo-relevance feedback

Download Axiomatic Analysis of Smoothing Methods in Language Models for Pseudo-Relevance Feedback

Post on 18-Feb-2017

344 views

Category:

Data & Analytics

1 download

Embed Size (px)

TRANSCRIPT

  • 10/1/2015

    Axiomatic Analysis of Smoothing Methods in Language Models for Pseudo-Relevance Feedback

    HUSSEIN HAZIMEH AND CHENGXIANG ZHAI

    UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN

    1

  • Pseudo Relevance Feedback

    Judgments:

    d1 +

    d2 -

    d3 +

    dk -

    ...

    Query Retrieval

    Engine

    Results:

    d1 3.5

    d2 2.4

    dk 0.5

    ...

    User

    Document

    collection

    Judgments:

    d1 +

    d2 +

    d3 +

    dk -

    ...

    top 10

    Pseudo feedback

    Assume top 10 docsare relevant

    Relevance feedback User judges documents

    New

    Query

    FeedbackLearn from

    Examples

    2

  • Pseudo-Relevance Feedback

    Its blind!

    Good for high recall information needs

    A Blind Superhero. Courtesy of iStock

    3

  • Collection-based Smoothing

    Collection-based smoothing is generally used for LM-based retrieval functions and for PRF models

    A commonly used collection-based smoothing scheme is Dirichletprior smoothing:

    Dirichlet Prior (Smoothing Parameter)

    Document Length

    Count of Word in Document

    4

  • Study of Smoothing Methods in PRF

    We will establish both analytically and empirically that collection-based smoothing is not a good choice for PRF: It forces PRF models to select very common words

    Additive smoothing will be shown to outperform the collection-based counterpart

    5

  • How Do LM PRF Models Work?

    D1

    Dn

    AveragingFunction:

    Scoring

    Function:

    (|1)

    (|)

    (|)

    (|)

    6

  • How Do LM PRF Models Work?

    The feedback LM, , would generally have the following form:

    : is an averaging function, e.g. geometric mean

    :2 is a function increasing in the first argument and decreasing in the second

    Rewards common words in feedback set

    Penalizes common words in collection

    7

  • Problem!

    The first argument rewards common words in the collection while the second penalizes them. The analysis shows that the first argument usually wins!

    Rewards common words in feedback set

    and collection

    Penalizes common words in collection

    Proportional to (|)

    8

  • Overview of the Analysis

    We considered three PRF models in the study: Divergence Minimization Model

    Relevance Model

    Geometric Relevance Model

    Next, we will briefly discuss how the DMM and GRM work and then give an overview of the axiomatic analysis.

    The analysis of the RM is very similar to the GRM and the same results apply

    9

  • Divergence Minimization Model (Zhai and Lafferty, 2001)

    The DMM solves the following optimization problem:

    The solution has a closed form and is given by:

    10

  • Geometric Relevance Model (Seo and Croft, 2010)

    An enhanced form of the Relevance Model (RM) that replaces the arithmetic mean used in RM by the geometric mean:

    Note that the function above is not is not affected by (|), i.e., the model is not designed to penalize common words.

    11

  • Main Axiom: IDF Effect (Clinchant and Gaussier, 2013)

    Rationale: A PRF model is expected to penalize common words in the collection in order to select high quality discriminative terms.

    Given any two words 1and 2 from the feedback set 1, 2,

    12

  • DMM with Collection-based smoothing: IDF Effect

    Study the sign of:

    Not straightforward. Strategy: Find an attainable lower bound on the expression above

    Study the sign of the lower bound

    If the lower bound is strictly positive, then DMM supports the IDF effect

    13

  • DMM: Results of Analysis

    Conclusion: Using collection-based smoothing the DMM will be either consistently reward common terms or will select only one feedback term

    14

  • GRM with Collection-based smoothing: IDF Effect

    The GRM cannot support the IDF effect:

    It consistently rewards favors common words in the collection

    15

  • Proposed Solution: Additive Smoothing

    Words get additional pseudo-counts:

    Next, we show how additive smoothing prevents the models from rewarding common terms

    16

  • DMM with Additive Smoothing: IDF Effect

    The DMM unconditionally supports the IDF Effect:

    Now it is performing the intended objective!

    17

  • DMM: Empirical Validation

    Query: Computer

    18

  • GRM with Additive Smoothing: IDF Effect

    Although the IDF effect is still not supported:

    However, common terms are no longer being rewarded!

    19

  • GRM: Empirical Validation

    Query: Computer

    20

  • Empirical Evaluation: Retrieval Measures

    21

  • Empirical Evaluation: Robustness of Additive Smoothing

    22

  • Measuring the Discrimination of PRF Models

    In previous studies, the average of the IDF of the top terms was used as an indicator of how discriminative the terms selected by a PRF method are

    Such a measure might not work well in some cases

    We propose the Discrimination Measure (DM):

    Expected Document Frequency

    Constant

    23

  • Empirical Evaluation: Discrimination Measure

    A several-fold decrease in the expected document frequency

    24

  • Conclusion

    Collection-based smoothing forces PRF models to select very common terms The same problem might exist in other applications where LMs are aggregated

    Additive smoothing prevents PRF models from rewarding common terms and increases the retrieval performance significantly

    A new measure for quantifying PRF Discrimination

    25

  • Future Work

    Should PRF models penalize common words?

    Analysis of other smoothing methods such as topic-based smoothing

    Inspect areas, other than PRF, where collection-based smoothing is used in aggregating language models

    26

  • Thanks to SIGIR for the Student Travel Grant!

    Thank you for Listening!

    27

View more >