Current Approaches in Search Result Diversification

Mario Sangiorgio

Presentation outline

Problem definition

What is diversity?

The relevance/diversity trade-off

Performance evaluation

Open issues and conclusions

Why is result diversification needed?

A couple of real life examples

Flash

Ambiguous query

Unambiguous query

Nuclear power plant

Search result diversification is an optimization

problem aiming to find k items which are the

subset of all relevant results that contains both

most relevant and most diverse results.

Problem definition

What is needed

Relevance measure Diversity measure

Diversification objective

The result diversification process

Items are ranked by relevance Diversity is measured

The two measures are used to get the final ranking

What is diversity?

How can items be diverse?

Word sense diversity, from ambiguous queries

Information source diversity, from

unambiguous queries

Measures of diversity

Diversity is tightly coupled with the concept of similarity

To address the different aspects of the problem several measures emerged:

Semantic distanceCategorical distance

Novel information

Semantic distanceDiversifies on content dissimilarity

Uses the min-hashing scheme to get the

sketch of a document

sim u , v =∣Su ∩Sv ∣∣Su ∪Sv ∣Distance is computed

from Jaccard similarity d u , v=1−sim u , v

Sd ={MH h1 d , ... , MH hnd }

Does not work well when the documents have too different lengths or small sketch size

Categorical distanceEmphasizes word sense diversification

It is based on metadata (Taxonomy)

d u , v= ∑i=lca u ,v

l u 1

2e i−1 ∑i=lca u ,v

l v 1

2e i−1

Examples of taxonomies:/Top/Health vs /Top/Finance

/Top/Sport/Racing vs /Top/Sport/Football

The measure is a weighted tree distance

Novel information

Results are represented with unigram language models (Used for natural language processing)

Diversifies on a general sense regarding content dissimilarity. Good for subtopics

For each document is evaluated (with the Kullback-Leibler divergence) how much novel

information it brings into the set

How many extra bits will be needed to describe the new document using only the already

selected document in the set

Diversity measures: open issues

Some aspects not taken into account:

intrinsic properties of the document

genre of the document

sentiment regarding the topic

The relevance/diversity optimization problem

Diversification objectivesIt has been proved impossible to find a function

that has all the required properties:

scale invarianceconsistency

richness stability

independence of irrelevant attributesmonotonicity

strength of relevancestrength of similarity

Diversification objectivesSeveral functions proposed:

Max sum(No stability)

Max min(No consistency

nor stability)

Mono objective(No consistency)

Max sum of max score(Maximizes relevancy and

then diversity)

Max product(It is based on the already

chosen results)

Categorical(Results have to cover

a set of categories)

Diversification algorithmsFinding the best solution is a NP-Hard problem

Algorithm depends on the objective function

Approximation Greedy

Open issues:

Is Off-linepre-computation

applicable?

Are there efficient data structures?

How to evaluate diversity in search

Full textTREC Interactive

Top results from commercial search engine

Structured dataTaxonomies (Open Directory Project)

Ground truthWikipedia disambiguation pages

Judgements from Amazon Mechanical Turk

There is the need of task-specific standard datasets

Data set for the evaluation

Benchmarks

Adaptation from existing metrics:

Alpha-NDCGNormalized discounted

cumulative gain

Subtopic recall and precision

Number of subtopics covered

User intentResults distribution

should reflect what the user is asking for

Comparison against the optimum

Alpha-nDCG

Based on information nuggets (Answer to a question)

A document is relevant when it contains a nugget needed by the user

Quality of results graded by human assessors

The most nuggets are in the set the best

Subtopic recall and precision

s−recall at k=number of subtopics covered by the first k documentstotal number of subtopics

s− precision at r=minRank S opt , r minRank S , r

Is the result set exhaustive?

Is the result set efficient?

Conclusions

Diversification can really improve quality of search results

There is still some work to do in order to achieve good results in all the possible scenarios

There is room for improvement defining new diversity types and metrics

Ranking functions should take in account diversity from the beginning in an integrated process

Datasets to evaluate each notion of diversity should be built

Open issues

References Minack, E., Demartini, G., Nejdl W.: Current Approaches to Search

Result Diversification. In: Proceedings of ISWC '09

Gollapudi, S., Sharma, A.: An Axiomatic Approach for Result Diversification.In: Proocedings of WWW '09

Zhai, C.X., Cohen, W.W., Lafferty, J.: Beyond Independent Relevance: Methods and Evaluation Metrics for Subtopic Retrieval. In: Proceedings

of SIGIR '03

Agrawal, R., Gollapudi, S., Halverson, A., Ieong, S.: Diversifying Search Results. In: Proceedings of WSDM '09

Clough, P., Sanderson, M., Abouammoh, M., Navarro, S., Paramita, M.: Multiple Approaches to Analysing Query Diversity. In: Proceedings of

SIGIR '09

Clarke, C.L., Kolla, M., Cormack, G.V., Vechtomova, O., Ashkan, A., Büttcher, S., MacKinnon, I.: Novelty and Diversity in Information

Retrieval Evaluation. In: Proceedings of SIGIR '08