A Machine Learning Approach to Sentence Ordering for Multidocu

ment Summarization and Its Evaluation

D. Bollegala, N. Okazaki and M. IshizukaThe University of Tokyo

IJCNLP 2005

2/21

Abstract

• Ordering information is a difficult but an important task for natural language generation applications

• This paper proposes an algorithm that learns orderings from a set of human ordered texts

• Our model consists of a set of ordering experts which give their precedence preference between two sentences

• Our experimental results show that the proposed algorithm outperforms the existing methods in all evaluation metrics

3/21

Introduction

• The task of ordering sentences arises in many fields– Multidocument summarization (MDS), question and answering, e

tc.

• Two stages of MDS– The source documents are analyzed and a set of sentences are

extracted– Create a coherent summary from the extracts

• Sentence ordering has received lesser attention comparing to the first stage– Chronological ordering (Barzilay et al., 2002)– Topical relevance ordering (Barzilay et al., 2002)– Probabilistic model of text structuring (Lapata, 2003)– Precedent relation among sentences (Okazaki, 2005)

4/21

Introduction

• The appropriate way to combine these different methods to obtain more robust and coherent text remains unknown

• In this paper, we learn the optimum linear combination of these heuristics that maximizes readability of a summary using a set of human-made orderings

• We propose two new evaluation metrics– Weighted Kendall Coefficient– Average Continuity

5/21

Method

• To decide the order among sentences, we implement five ranking experts– Chronological– Probabilistic– Topical relevance– Precedent– Succedent

• Each expert e generates a pair-wise preference function defined as following (Cohen, 1999)

6/21

Chronological Expert

• Chronological expert emulates conventional chronological ordering

T(u): publication date of sentence uD(u): the unique identifier of the document to which u belongsN(u): the line number of sentence u in the original document

7/21

Probabilistic Expert

• Probabilistic model (Lapata, 2003)

8/21

Probabilistic Expert

• Only nouns and verbs as features• Back-off smoothing• Probabilistic expert

r : the lastly ordered sentence in Q

9/21

Topical Relevance Expert

• Grouping the extracted sentences which belong to the same topic, improves readability of the summary– Use cosine similarity

10/21

Precedent Expert

• When placing a sentence in the summary it is important to check whether the preceding sentences convey the necessary background information

11/21

Succedent Expert

12/21

Ordering Algorithm

• Find the optimal order of a given total preference function is NP-complete

• Greedy algorithm (Cohen,1999)

13/21

Learning Algorithm

• Weighted allocation algorithm (Cohen, 1999)

Wi1 =0.2

5.0

14/21

Evaluation

• In addition to Kendall’s coefficient and Spearman’s rank correlation coefficient, we use sentence continuity and two new proposed metrics– Weighted Kendall– Average Continuity

• Human Objects

15/21

Evaluation

• Kendall’s coefficient

– Major drawback• It does not take into consideration the relative distance d

between the discordant pairs

• Weighted Kendall Coefficient

d: the number of sentences that lie between the discordant pair

Q: the number of discordant pairs

16/21

Evaluation

• Continuity metric expresses the continuity of the sentences (Readability)

• Average Continuity– Sentence block (sentence n-gram)

17/21

Results

• 3rd Text Summarization Challenge (TSC) corpus– News articles from Mainichi and Yomiuri– 30 different topics– Create 30 summaries by ordering the

extraction data of TSC-3 by hand– 10-fold validation– Random order (RO), Probabilistic Ordering

(PO), Chronological Ordering (CO), Learned Ordering (LO), Human-made Ordering (HO)

18/21

Results

19/21

Results

20/21

Result of Human Evaluation

Two human judges

21/21

Conclusion

• Our method integrates all the existing approaches to sentence ordering

• The results reveal that our algorithm for sentence ordering did contribute to summary readability

• In the future,– We plan to further study on the sentence ordering

problem– Extending our algorithm to other natural language

generation application