the optimum clustering framework: implementing the cluster hypothesis

A Framework for Optimum DocumentClustering:

Implementing the Cluster Hypothesis

Norbert Fuhr

University of Duisburg-Essen

March 30, 2011

A Framework for Optimum Document Clustering: Implementing the Cluster Hypothesis 2

Outline

1 Introduction

2 Cluster Metric

3 Optimum clustering

4 Towards Optimum Clustering

5 Experiments

6 Conclusion and Outlook

Introduction

1 Introduction

2 Cluster Metric

5 Experiments

Introduction

Motivation

Ad-hoc Retrievalheuristic models:

define retrieval functionevaluate to test if it yields good quality

Probability Ranking Principle (PRP)theoretic foundation for optimum retrievalnumerous probabilistic models based on PRP

Document clusteringclassic approach:

define similarity function and fusion principleevaluate to test if they yield good quality

Optimum Clustering Principle?

Introduction

Motivation

Introduction

Motivation

Introduction

Cluster Hypothesis

Original Formulation”closely associated documents tend to be relevant to the samerequests” (Rijsbergen 1979)

Idea of optimum clustering:

Cluster documents in such a way, that for any request, therelevant documents occur together in one cluster

redefine document similarity:documents are similar if they are relevant to the same queries

Introduction

Cluster Hypothesis

Introduction

Cluster Hypothesis

Introduction

The Optimum Clustering Framework

Introduction

Cluster Metric

1 Introduction

2 Cluster Metric

5 Experiments

Cluster Metric

Defining a Metric based on the Cluster Hypothesis

General idea:Evaluate clustering wrt. a set of queriesFor each query and each cluster, regard pairs ofdocuments co-occurring:

relevant-relevant: goodrelevant-irrelevant: badirrelevant-irrelevant: don’t care

Cluster Metric

Pairwise precision

Q Set of queriesD Document collectionR relevance judgments: R ⊂ Q × DC Clustering, C = {C1, . . . ,Cn} s.th. ∪n

i=1Ci = D and∀i , j : i 6= j → Ci ∩ Cj = ∅

ci = |Ci | (size of cluster Ci ),rik = |{dm ∈ Ci |(qk ,dm) ∈ R}| (number of relevant

documents in Ci wrt. qk )

Pairwise precision (weighted average over all clusters)

Pp(D,Q,R, C) =1|D|

∑Ci∈Cci>1

∑qk∈Q

rik (rik − 1)ci(ci − 1)

Cluster Metric

Pairwise precision

Pp(D,Q,R, C) =1|D|

∑Ci∈Cci>1

∑qk∈Q

Cluster Metric

Pairwise precision

Pp(D,Q,R, C) =1|D|

∑Ci∈Cci>1

∑qk∈Q

Cluster Metric

Pairwise precision

Pp(D,Q,R, C) =1|D|

∑Ci∈Cci>1

∑qk∈Q

Cluster Metric

Pairwise precision

Pp(D,Q,R, C) =1|D|

∑Ci∈Cci>1

∑qk∈Q

Cluster Metric

Pairwise precision – Example

Pp(D,Q,R, C) =1|D|

∑Ci∈Cci>1

∑qk∈Q

Query set: disjoint classification with two classes a and b,three clusters: (aab|bb|aa)Pp = 1

7(3(13 + 0) + 2(0 + 1) + 2(1 + 0)) = 5

Perfect clustering for a disjoint classification would yieldPp = 1for arbitrary query sets, values > 1 are possible

Cluster Metric

Pairwise precision – Example

Pp(D,Q,R, C) =1|D|

∑Ci∈Cci>1

∑qk∈Q

Query set: disjoint classification with two classes a and b,three clusters: (aab|bb|aa)Pp = 1

7(3(13 + 0) + 2(0 + 1) + 2(1 + 0)) = 5

Perfect clustering for a disjoint classification would yieldPp = 1for arbitrary query sets, values > 1 are possible

Cluster Metric

Pairwise recall

rik = |{dm ∈ Ci |(qk ,dm) ∈ R}| (number of relevantdocuments in Ci wrt. qk )

gk = |{d ∈ D|(qk ,d) ∈ R}| (number of relevantdocuments for qk )

(micro recall)

Rp(D,Q,R, C) =∑

qk∈Q∑

Ci∈C rik (rik − 1)∑qk∈Qgk>1

gk (gk − 1)

Example: (aab|bb|aa)2 a pairs (out of 6)1 b pair (out of 3)Rp = 2+1

6+3 = 13 .

Cluster Metric

Pairwise recall

rik = |{dm ∈ Ci |(qk ,dm) ∈ R}| (number of relevantdocuments in Ci wrt. qk )

gk = |{d ∈ D|(qk ,d) ∈ R}| (number of relevantdocuments for qk )

(micro recall)

Rp(D,Q,R, C) =∑

qk∈Q∑

gk (gk − 1)

Example: (aab|bb|aa)2 a pairs (out of 6)1 b pair (out of 3)Rp = 2+1

6+3 = 13 .

Cluster Metric

Perfect clustering

C is a perfect clustering iff there exists no clustering C′ s.th.Pp(D,Q,R, C) < Pp(D,Q,R, C′)∧Rp(D,Q,R, C) < Rp(D,Q,R, C′)

strong Pareto optimum – more than one perfect clusteringpossible

Example:

Pp({d1,d2,d3}, {d4,d5}) =Pp({d1,d2}, {d3,d4,d5}) = 1,Rp = 2

Pp({d1,d2,d3,d4,d5}) = 0.6,Rp = 1

Cluster Metric

Perfect clustering

Example:

Pp({d1,d2,d3}, {d4,d5}) =Pp({d1,d2}, {d3,d4,d5}) = 1,Rp = 2

Pp({d1,d2,d3,d4,d5}) = 0.6,Rp = 1

Cluster Metric

Perfect clustering

Example:Pp({d1,d2,d3}, {d4,d5}) =Pp({d1,d2}, {d3,d4,d5}) = 1,Rp = 2

Pp({d1,d2,d3,d4,d5}) = 0.6,Rp = 1

Cluster Metric

Do perfect clusterings form a hierarchy?

Cluster Metric

C = {{d1,d2,d3,d4}}

Cluster Metric

C = {{d1,d2,d3,d4}}

C′ = {{d1,d2}, {d3,d4}}

Cluster Metric

CC’’

C = {{d1,d2,d3,d4}}

C′ = {{d1,d2}, {d3,d4}} C′′ = {{d1,d2,d3}, {d4}}

Optimum clustering

1 Introduction

2 Cluster Metric

5 Experiments

Optimum clustering

Optimum Clustering

Usually, clustering process has no knowledge aboutrelevance judgmentsswitch from external to internal cluster measuresreplace relevance judgments by estimates of probability ofrelevancerequires probabilistic retrieval method yielding P(rel |q,d) compute expected cluster quality

Optimum clustering

Optimum Clustering

Optimum clustering

Optimum Clustering

Optimum clustering

Optimum Clustering

Optimum clustering

Optimum Clustering

Optimum clustering

Expected cluster quality

Pairwise precision:

Pp(D,Q,R, C) =1|D|

∑Ci∈Cci>1

∑qk∈Q

Expected precision:

!π(D,Q, C) = 1|D|

∑Ci∈C|Ci |>1

ci(ci − 1)

∑qk∈Q

∑(dl ,dm)∈Ci×Ci

dl 6=dm

P(rel |qk ,dl)P(rel |qk ,dm)

Optimum clustering

Expected cluster quality

Pairwise precision:

Pp(D,Q,R, C) =1|D|

∑Ci∈Cci>1

∑qk∈Q

Expected precision:

!π(D,Q, C) = 1|D|

∑Ci∈C|Ci |>1

ci(ci − 1)

∑qk∈Q

dl 6=dm

Optimum clustering

Expected precision

π(D,Q, C) = 1|D|

∑Ci∈C|Ci |>1

ci(ci − 1)

dl 6=dm

∑qk∈Q

here∑

qk∈Q P(rel |qk ,dl)P(rel |qk ,dm) gives the expectednumber of queries for which both dl and dm are relevant

Transform a document into a vector of relevance probabilities:~τT (dm) = (P(rel |q1,dm),P(rel |q2,dm), . . . ,P(rel |q|Q|,dm)).

π(D,Q, C) =1|D|

∑Ci∈C|Ci |>1

1ci − 1

dl 6=dm

~τT (dl) · ~τ(dm)

Optimum clustering

Expected precision

π(D,Q, C) = 1|D|

∑Ci∈C|Ci |>1

ci(ci − 1)

dl 6=dm

∑qk∈Q

here∑

qk∈Q P(rel |qk ,dl)P(rel |qk ,dm) gives the expectednumber of queries for which both dl and dm are relevant

Transform a document into a vector of relevance probabilities:~τT (dm) = (P(rel |q1,dm),P(rel |q2,dm), . . . ,P(rel |q|Q|,dm)).

π(D,Q, C) =1|D|

∑Ci∈C|Ci |>1

1ci − 1

dl 6=dm

Optimum clustering

Expected recall

Rp(D,Q,R, C) =∑

qk∈Q∑

gk (gk − 1)

Direct estimation requires estimation of denominator→ biasedestimatesBut: denominator is constant for a given query set→ ignore

compute an estimate for the numerator only:

ρ(D,Q, C) =∑Ci∈C

dl 6=dm

(Scalar product ~τT (dl) · ~τ(dm) gives the expected number ofqueries for which both dl and dm are relevant)

Optimum clustering

Expected recall

Rp(D,Q,R, C) =∑

qk∈Q∑

gk (gk − 1)

dl 6=dm

Optimum clustering

Expected recall

Rp(D,Q,R, C) =∑

qk∈Q∑

gk (gk − 1)

dl 6=dm

Optimum clustering

Expected recall

Rp(D,Q,R, C) =∑

qk∈Q∑

gk (gk − 1)

dl 6=dm

Optimum clustering

C is an optimum clustering iff there exists no clustering C′ s.th.π(D,Q, C) < π(D,Q, C′) ∧ ρ(D,Q, C) < ρ(D,Q, C′)

Pareto optimaSet of perfect (and optimum) clusterings not even forms acluster hierarchy no hierarchic clustering method will find all optima!

Optimum clustering

Towards Optimum Clustering

1 Introduction

2 Cluster Metric

5 Experiments

Towards Optimum ClusteringDevelopment of an (optimum) clustering method

1 Set of queries,2 Probabilistic retrieval method,3 Document similarity metric, and4 Fusion principle.

A simple application

1 Set of queries: all possible one-term queries2 Probabilistic retrieval method: tf ∗ idf3 Document similarity metric: ~τT (dl) · ~τ(dm)

4 Fusion principle: group average clustering

π(D,Q,C) =1

c(c − 1)

∑(dl ,dm)∈C×C

dl 6=dm

standard clustering method

π(D,Q,C) =1

c(c − 1)

∑(dl ,dm)∈C×C

dl 6=dm

π(D,Q,C) =1

c(c − 1)

∑(dl ,dm)∈C×C

dl 6=dm

π(D,Q,C) =1

c(c − 1)

∑(dl ,dm)∈C×C

dl 6=dm

π(D,Q,C) =1

c(c − 1)

∑(dl ,dm)∈C×C

dl 6=dm

π(D,Q,C) =1

c(c − 1)

∑(dl ,dm)∈C×C

dl 6=dm

π(D,Q,C) =1

c(c − 1)

∑(dl ,dm)∈C×C

dl 6=dm

Query set

Too few queries in real collections→ artificial query set

collection clustering: set of all possible one-term queriesProbability distribution over the query set: uniform /proportional to doc. freq.Document representation: original terms / transformationsof the term spaceSemantic dimensions: focus on certain aspects only (e.g.images: color, contour, texture)

result clustering: set of all query expansions

Probabilistic retrieval method

Model: In principle, any retrieval model suitableTransformation to probabilities: direct estimation /transforming the retrieval score into such a probability

Document similarity metric.

fixed as ~τT (dl) · ~τ(dm)

Fusion principles

OCF only gives guidelines for good fusion principles:consider metrics π and/or ρ during fusion

Group average clustering:

σ(C) =1

c(c − 1)

∑(dl ,dm)∈C×C

dl 6=dm

expected precision as criterion!starts with singleton clusters minimum recallbuilding larger clusters for increasing recallforms cluster with highest precision(which may be lower than that of the current clusters)

σ(C) =1

c(c − 1)

∑(dl ,dm)∈C×C

dl 6=dm

σ(C) =1

c(c − 1)

∑(dl ,dm)∈C×C

dl 6=dm

σ(C) =1

c(c − 1)

∑(dl ,dm)∈C×C

dl 6=dm

Fusion principles – min cut

starts with single cluster (maximum recall)searches for cut with minimum loss in recall

∑(dl ,dm)∈C×C

dl 6=dm

consider expected precision for breaking ties!

∑(dl ,dm)∈C×C

dl 6=dm

∑(dl ,dm)∈C×C

dl 6=dm

Finding optimum clusterings

Min cut(assume cohesive similarity graph)

starts with optimum clustering for maximum recallmin cut finds split with minimum loss in recallconsider precision for tie breaking optimum clustering for two clustersO(n3) (vs. O(2n) for the general case)subsequent splits will not necessarily reach optima

Group averagein general, multiple fusion steps for reaching first optimumgreedy strategy does not necessarily find this optimum!

Experiments

1 Introduction

2 Cluster Metric

5 Experiments

Experiments

Experiments with a Query Set

ADI collection:35 queries70 documents (relevant to 2.4 queries on avg.)

Experiments:Q35opt using the actual relevance in ~τ(d)

Q35 BM25 estimates for the 35 queries1Tuni 1-term queries, uniform distribution1Tdf 1-term queries, according to document frequency

Experiments

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Recall

Q35optQ35

1Tuni1Tdf

Experiments

Using Keyphrases as Query SetCompare clustering results based on different query sets

1 ‘bag-of-words’: single words as queries2 keyphrases automatically extracted as head-noun phrases,

single query = all keyphrases of a document

Test collections:4 test collections assembled from the RCV1 (Reuters)news corpus# documents: 600 vs. 6000# categories: 6 vs. 12,Frequency distribution of classes: ([U]niform vs.[R]andom).

Experiments

Using Keyphrases as Query SetCompare clustering results based on different query sets

1 ‘bag-of-words’: single words as queries2 keyphrases automatically extracted as head-noun phrases,

single query = all keyphrases of a document

Test collections:4 test collections assembled from the RCV1 (Reuters)news corpus# documents: 600 vs. 6000# categories: 6 vs. 12,Frequency distribution of classes: ([U]niform vs.[R]andom).

Experiments

Using Keyphrases as Query Set - Results

Average Precision (External) F-measure

Experiments

Evaluation of the Expected F-Measure

Correlation between expected F-Measure (internal measure)andstandard F-measure (comparison with reference classification)

test collections as beforeregard quality of 40 different clustering methods for eachsetting(find optimum clustering among these 40 methods)

Experiments

Correlation resultsPearson correlation between internal measures and theexternal F-Measure

Conclusion and Outlook

1 Introduction

2 Cluster Metric

5 Experiments

Summary

Optimum Clustering Framework

makes Cluster Hypothesis a requirementforms theoretical basis for development of better clusteringmethodsyields positive experimental evidence

Further Research

theoreticalcompatibility of existing clustering methods with OCFextension of OCF to soft clusteringextension of OCF to hierarchical clustering

experimentalvariation of query setsuser experiments

the optimum clustering framework: implementing the cluster hypothesis

Technology

knowledge aboutrelevance judgmentsswitch

dene similarity function

dene retrieval functionevaluate

multiple fusion steps

hierarchic clustering method

group average clustering1

query sets1 bag

arbitrary query sets