1

Opinion Summarization Using En-tity Features and Probabilistic

Sentence Coherence Optimization

(UIUC at TAC 2008 Opinion Summarization Pilot)

Nov 19, 2008

Hyun Duk Kim, Dae Hoon Park, V.G.Vinod Vydiswaran, ChengXiang Zhai

Department of Computer Science

University of Illinois, Urbana-Champaign

Research Questions

1. Can we improve sentence re-trieval by assigning more weights to entity terms?

2. Can we optimize the coherence of a summary using a statistical coherence model?

2

Opinionated Relevant Sentences

SentenceFiltering

Step 2

General Approach

3

Target1001

Q1

Q2

…

OpinionSummary

SentenceOrganization

Step 3

Doc

Query 1

Query 2

Relevant SentencesSentence

Retrieval

Step 1

?

Step 1: Sentence Retrieval

Target1001

Question i

Doc

Indri Toolkit

Relevant Sentences

Named Entity: 10Noun phrase : 2Others : 1

Uniform Term Weighting

Non-Uniform Weighting

Step 2: Sentence Filtering

5

RelevantSentences

S1’S2’S3’S1’S2’S4’S6’S7’S4’

S1’S2’S3’S1’S2’S4’S6’S7’S4’

Keep only thesame polarity

S1’S2’S3’S1’S2’S4’S6’S7’S4’

Keep only Opinionated

Sentences

Remove redundancy

S1’S2’S3’S1’S2’S4’S6’S7’S4’

Step 3: Summary Organization (Method 1: Polarity Ordering)

– Paragraph structure by question and polarity

– Add guiding phrase

6

The first question is … Following are positive opinions…

Following are negative opinions…

The second question is … Following are mixed opinions… …

Step 3: Summary Organization (Method 2: Statistical Coherence Optimization)

7

S1

S2

Sn

…

S1’

S2’

Sn’

…

S3 S3’

c(S1’, S2’)

c(S2’, S3’)

c(Sn-1’, Sn’)+

…

• Coherence function: c(Si, Sj)

• Use a greedy algorithm to order sentences to maximize the total score

c(S1’, S2’)+c(S1’, S2’)+…+c(Sn-1’, Sn’)

Probabilistic Coherence Function(Idea similar to [Lapata 03])

8

0.1)()(

001.0)"("),(ˆ

||||

),(),(

,

vcountucount

sentencesadjacenttwoinvanducountvup

ss

vupssc

ji svsuji

ji

Sentence 1

Sentence 2

u

v v

v

vu

u

Yes YesNoNoNo

P(u,v) = (2+0.001)/(3*4+1.0)

Average coherence probability (Pointwise mutual information)

over all word combinations

Train with original document

Opinionated Relevant Sentences

SentenceFiltering

Step 2

General Approach

9

Target1001

Q1

Q2

…

OpinionSummary

SentenceOrganization

Step 3

Query 1

Doc

SentenceRetrieval

Relevant Sentences

Step 1

Query 2

?

Opinionated Relevant Sentences

SentenceFiltering

Step 2

Submissions: UIUC1, UIUC2

10

Target1001

Q1

Q2

…

OpinionSummary

SentenceOrganization

Step 3

Query 1

Doc

SentenceRetrieval

Relevant Sentences

Step 1

Query 2

?

UIUC1: non-uniform weightingUIUC2: uniform weighting

UIUC1: Aggressive polarity filteringUIUC2: Conservative filtering

UIUC1: Polarity orderingUIUC2: Statistical ordering

Evaluation

• Rank among runs without answer-snippet

11

F-Score Grammatical-ity

Non-redun-dancy

Structure/Coherence

Fluency/Readability

Responsiveness

UIUC1Polarity 6 15 15 5 6 8

UIUC2Coherence 3 9 10 15 16 4

(Total: 19 runs)

Evaluation

• Rank among runs without answer-snippet

12

F-Score Grammatical-ity

Non-redun-dancy

Structure/Coherence

Fluency/Readability

Responsiveness

UIUC1Polarity 6 15 15 5 6 8

UIUC2Coherence 3 9 10 15 16 4

Statistical orderingNothing

(Total: 19 runs)

NE/NP retrieval, Polarity filtering

Polarity ordering

Evaluation of Named Entity Weight-ing

Target1001

Question i

Doc

Indri Toolkit

Relevant Sentences

Named entity: 10Noun phrase : 2Others : 1

Uniform Term Weighting

Non-Uniform Weighting

Assume a sentence is relevant iff similarity(sentence, nugget description) > threshold

Effectiveness of Entity Weighting

14

10-2-1Weighting

1-1-1Weighting

Polarity Module• Polarity module performance evaluation

on the sentiment corpus. [Hu&Liu 04, Hu&Liu 04b]

15

Classification result Positive Negative

NonOpinionated 1063 598

Positive 1363 371

Negative 383 412

Mixed 296 210

Total 3105 1591

Exact Match 1363/3105=0.44 412/1591=0.26

(1363+412)/(3105+1591)=0.38

Exact Opposite 383/3105=0.12 371/1591=0.23

(383+371)/(3105+1591)=0.16

(Unit: # of sentence)

Coherence optimization

• Evaluation methods– Basic assumption

• the sentence order of original document is coherent

– Among given target documents,use 70% as training set, 30% as test set.

– Measurement: strict pair matching• # of correct sentence pair / # of total adjacent sentence pair

16

Probabilistic Coherence Function

17

ji svsuji

ji ss

vupssc

, ||||

),(),( Average coherence probability

over all word combinations

2)(""

1)"("),(ˆ

sizedictionarypairswordofcounttotal

sentencesadjacenttwoinvanducountvup

0.1)()(

001.0)"("),(ˆ

vcountucount

sentencesadjacenttwoinvanducountvup

Point-wise Mutual information with smoothing

Strict joint probability

Probabilistic Coherence Function

18

))()(

),(log(),()

)()(

),(log(),(

))()(

),(log(),()

)()(

),(log(),(),(ˆ

vnotpunotp

vnotunotpvnotunotp

vnotpup

vnotupvnotup

vpunotp

vunotpvunotp

vpup

vupvupvup

Mutual information

where,

,1

25.0),(),(,

1

25.0),(),(

,1

25.0),(),(,

1

25.0),(),(

N

vnotunotcvnotunotp

N

vnotucvunotp

N

vunotcvunotp

N

vucvup

N = c(u,v)+c(not u, v)+c(u, not v)+c(not u, not v)For unseen pairs, p(u,v)=0.5*MIN(seen pairs in training)

Coherence optimization test

– Pointwise mutual information effectively penalize common words

19

Selection of training words Strict Joint Probability

MutualInformation

PointwiseMutual

Information

No Omission 0.022259 0.041651 0.056063

Omitted stopwords 0.031389 0.054554 0.057119

Omitted frequent words (counts > 33) (counts > 11) (counts > 6) (counts > 2)

0.0313890.0270130.0204480.019769

0.0514600.0457250.0322190.022259

0.0494980.0461030.0347850.021882

Omitted rare words (counts < 33) (counts < 14) (counts < 6)

(counts < 2)

0.0222590.0220330.0212030.022259

0.0328980.0328230.0370480.041802

0.0440650.0490450.0549310.056591

Coherence optimization test• Top ranked p(u,v) of strict joint probability

– A lot of stopwords are top-ranked.

20

u v p(u,v)

theto

thethetheof

anda

the…

thethetoof

andthethethea…

1.75E-0031.22E-0031.21E-0031.14E-0031.14E-0031.13E-0031.12E-0031.06E-0031.06E-003

…

Coherence optimization test

– Pointwise Mutual information was better than joint probability and normal mutual information.

– Eliminating common words, very rare words improved per-formance

21

Selection of training words Coherence score

Baseline: random order 0.01586

Strict Joint ProbabilityMutual Information

Pointwise Mutual Information (UIUC2)

0.043080.0416510.056063

Omitted stopwordsOmitted non-stopwords

0.0571190.020750

Omitted 95% least frequent words (counts < 33): Omitted 90% least frequent words (counts < 14): Omitted 80% least frequent words (counts < 6): Omitted 60% least frequent words (counts < 2):

0.0440650.0490450.0549310.056591

22

Conclusions

• Limited improvement in retrieval performance using named entity and noun phrase

• Need for a good polarity classification module

• Possibility on the improvement of statistical sentence ordering module with different co-herence function and word selection

Thank you

23

University of Illinois at Urbana-ChampaignHyun Duk Kim (hkim277@uiuc.edu)