1 ceratops center for extraction and summarization of events and opinions in text janyce wiebe, u....

1

CERATOPS Center for Extraction and Summarization

of Events and Opinions in Text

Janyce Wiebe, U. PittsburghClaire Cardie, Cornell U.

Ellen Riloff, U. Utah

2

Outline

• Sample of research results since March 2008– Coreference resolution– Topics and discourse

• Topic identification for fine-grained opinion analysis• Discourse-level opinion graphs for polarity classification

• Resources and systems– Upcoming releases– Jodange system for extracting and aggregating opinions from

on-line content

• Interactions • Research plans for the remainder of the grant

3

NP Coreference Resolution

Australian press has launched a bitter attack on Italy after seeing their beloved Socceroos eliminated on a controversial late penalty.

4

NP Coreference Resolution

Australian press has launched a bitter attack on Italy after seeing their beloved Socceroos eliminated on a controversial late penalty. Italian coach Lippi has also been blasted for his comments after the game.In the opposite camp Lippi is preparing his side for the upcoming game with Ukraine. He hailed 10-man Italy's determination to beat Australia and said the penalty was rightly given.

5

Coreference and Opinion Holders

Stoyanov & Cardie [2006]

Australian press has launched a bitter attack on Italy after seeing their beloved Socceroos eliminated on a controversial late penalty. Italian coach Lippi has also been blasted for his comments after the game.In the opposite camp Lippi is preparing his side for the upcoming game with Ukraine. He hailed 10-man Italy's determination to beat Australia and said the penalty was rightly given.

6

Reconcile

• NP coreference resolution toolbox/testbed– Public domain

• Joint effort of Cornell, LLNL, Univ of Utah• Algorithms for

– Training NP coref classifiers– Generating standard feature vectors– Testing/training on standard data sets (MUC, ACE)– Evaluating w.r.t. multiple metrics (MUC, B3, CEAF, etc.)

• Trained versions of state-of-the-art systems

7

Outline

• Sample of research results since March 2008– Coreference resolution– Topics and discourse

• Topic identification for fine-grained opinion analysis• Discourse-level opinion graphs for polarity classification

• Resources and systems– Upcoming releases– Jodange system for extracting and aggregating opinions from

on-line content

• Interactions • Research plans for the remainder of the grant

8

Topics in Fine-grained Opinion Analysis

– Opinion expression– Polarity

• positive• negative• neutral

– Strength/intensity• low..extreme

– Source (opinion holder)– Target (topic)

“The Australian Press launched a bitter attack on Italy”

Opinion FramePolarity: negative Intensity: highSource: “The Australian Press”Target: “Italy”

9

Opinion Summaries

Australian PressAustralian Press

ItalyMarcello Lippi

penalty

Socceroos

10

Definitions

• Topic - the real-world object, event or abstract entity that is the subject of the opinion as intended by the opinion holder

• Topic span - the closest, minimal span of text that mentions the topic

• Target span - the span of text that covers the syntactic surface form comprising the contents of the opinion

Stoyanov & Cardie [LREC 2008, Coling 2008]

11

Definitions

(1) [OH John] likes Marseille for its weather and cultural diversity.

(1) [OH John] likes [TARGET+TOPIC SPAN Marseille] for its weather and cultural diversity.

12

Definitions

(2) [OH Al] thinks that the government should tax more in order to curb CO2 emissions.

(2) [OH Al] thinks that [TARGET SPAN the government should tax gas more in order to curb CO2 emissions].

(2) [OH Al] thinks that [TARGET SPAN [TOPIC SPAN? the government] should [TOPIC SPAN? tax gas] more in order to [TOPIC SPAN? curb [TOPIC SPAN? CO2 emissions]]].

(3) Although he doesn’t like government imposed taxes, he thinks that a fuel tax is the only effective solution.

13

Issues in Opinion Topic Identification

• Multiple potential topics • Opinion topics are not always explicitly

mentioned

(3) [OH John] believes the violation of Palestinian human rights is one of the main factors.

Topic: ISRAELI-PALESTINIAN CONFLICT

(4) [OH I] disagree entirely!

14

A Coreference Approach

• Hypothesize that the notion of topic coreference will facilitate identification of opinion topics

• Easier than specifying the topic of each opinion in isolation

Two opinions are topic-coreferent if they share the same opinion topic.

15

A Coreference Approach to Opinion Topic Identification

• Find the clusters of topic-coreferent opinions– the critical step for topic identification

• Label the clusters with the name of the topic– we currently ignore this step – address in future work through frequency analysis of

the terms in each of the clusters

16

The Topic Coreference Algorithm

• Adapt a standard machine learning-based approach to NP coreference resolution (Soon et al., 2001; Ng & Cardie, 2002)1. identify topic spans2. perform pairwise classification of the associated

opinions and topic spans to determine whether or not they are topic-cofererent

3. cluster the opinions according to the results of (2)

17

Results• Annotated subset of the MPQA corpus for topic-

coreferent opinions– Acceptable inter-annotator agreement

• 10-fold cross-validation• Baselines

B3 CEAFall-in-one .37 -.10 .301 opinion per cluster .29 .22 .27same paragraph .55 .31 .50Choi 2000 .54 .37 .54

18

Results

sentence spans .57 .40 .54

automatic (heuristics) .57 .41 .54

modified manual topic spans .64 .51 .61

manual topic spans .71 .66 .62

B3 CEAFall-in-one .37 -.10 .301 opinion per cluster .29 .22 .27same paragraph .55 .31 .50Choi 2000 .54 .37 .54

19

Results

sentence spans .57 .40 .54automatic (heuristics) .57 .41 .54modified manual topic spans .64 .51 .61manual topic spans .71 .66 .62

B3 CEAFall-in-one .37 -.10 .301 opinion per cluster .29 .22 .27same paragraph .55 .31 .50Choi 2000 .54 .37 .54

20

Outline

• Sample of research results since March 2008– Coreference resolution– Topics and discourse

• Topic identification for fine-grained opinion analysis• Discourse-level opinion graphs for polarity classification

• Resources and systems– Upcoming releases– Jodange system for extracting and aggregating opinions from

on-line content

• Interactions • Research plans for the remainder of the grant

21

Discourse level opinion graphs for polarity classification

• Somasundaran et al. Discourse level Opinion Graphs for Polarity Classification. Submitted.

• Somasundaran et al. (2008). Discourse Level Opinion Interpretation. COLING-2008.

• Somasundaran et al. (2008). Discourse Level Opinion Relations: An Annotation Study. SIGdial Workshop on Discourse and Dialogue-2008.

22

Motivation

• The interpretations of opinions often depend on the other opinions in the discourse

23

Motivation• I like the LCD feature

• We must implement the LCD

• I think the LCD is hot

24

Motivation• I like the LCD feature

• We must implement the LCD

• I think the LCD is hot

Interdependent Interpretation of opinions in the discourse

25

• Shapes should be curved, so round shapes. Nothing square-like.

• ... So we shouldn’t have too square corners and that kind of

thing.

Motivation

26

• Shapes should be curved, so round shapes Nothing square-like.

• ... So we shouldn’t have too square corners and that kind of

thing.

Arguing Negative

Arguing Positive

Arguing Negative

Sentiment Negative

Motivation

27

• Shapes should be curved, so round shapes Nothing square-like.

• ... So we shouldn’t have too square corners and that kind of

thing.

Arguing Negative

Arguing Positive

Arguing Negative

Sentiment Negative

What were the opinions regarding the curved shape?Will the curved shape be accepted?

Motivation

28

• Shapes should be curved, so round shapes Nothing square-like.

• ... So we shouldn’t have too square corners and that kind of

thing.

Arguing Negative

Arguing Positive

Arguing Negative

Sentiment Negative

Direct opinion

Motivation

29

• Shapes should be curved, so round shapes Nothing square-like.

• ... So we shouldn’t have too square corners and that kind of

thing.

Arguing Negative

Arguing Positive

Arguing Negative

Sentiment Negative

Direct opinionOpinions towards mutually

exclusive option (alternative)

Motivation

30

• Shapes should be curved, so round shapes Nothing square-like.

• ... So we shouldn’t have too square corners and that kind of

thing.

Arguing Negative

Arguing Positive

Arguing Negative

Sentiment Negative

Direct opinionOpinions towards mutually

exclusive option (alternative)

Opinions towards the square shapes reveal more about the

speaker’s opinion regarding the curved shape

Motivation

31

• Shapes should be curved, so round shapes Nothing square-like.

• ... So we shouldn’t have too square corners and that kind of

thing.

Arguing Negative

Arguing Positive

Arguing Negative

Sentiment Negative

Direct opinionOpinions towards mutually

exclusive option (alternative)

More opinion information

Motivation

Opinions towards the square shapes reveal more about the

speaker’s opinion regarding the curved shape

32

Opinion Frames • This blue remote is cool.

• What’s more, the rubbery material is ergonomic.

• We must really go for the blue remote.

• I feel the red remote is a better choice.

• The blue remote will be too expensive.

Sentiment Pos

Sentiment Pos

Arguing Pos

Sentiment Pos

Sentiment Neg

33

Opinion Frames • This blue remote is cool.

• What’s more, the rubbery material is ergonomic.

• We must really go for the blue remote.

• I feel the red remote is a better choice.

• The blue remote will be too expensive.

Sentiment Pos

Sentiment Pos

Arguing Pos

Sentiment Pos

Sentiment Neg

same

34

Opinion Frames • This blue remote is cool.

• What’s more, the rubbery material is ergonomic.

• We must really go for the blue remote.

• I feel the red remote is a better choice.

• The blue remote will be too expensive.

Sentiment Pos

Sentiment Pos

Arguing Pos

Sentiment Pos

Sentiment Neg

same

same

35

Opinion Frames • This blue remote is cool.

• What’s more, the rubbery material is ergonomic.

• We must really go for the blue remote.

• I feel the red remote is a better choice.

• The blue remote will be too expensive.

Sentiment Pos

Sentiment Pos

Arguing Pos

Sentiment Pos

Sentiment Negalternative

same

same

37

Annotation tool

38

Opinion Frames • This blue remote is cool.

• What’s more, the rubbery material is ergonomic.

• We must really go for the blue remote.

• I feel the red remote is a better choice.

• The blue remote will be too expensive.

Sentiment Pos

Sentiment Pos

Arguing Pos

Sentiment Pos

Sentiment Negalternative

same

same

39

Opinion Frames • This blue remote is cool.

• What’s more, the rubbery material is ergonomic.

• We must really go for the blue remote.

• I feel the red remote is a better choice.

• The blue remote will be too expensive.

Sentiment Pos

Sentiment Pos

Arguing Pos

Sentiment Pos

Sentiment Neg

SPSPsame

alternative

same

same

41

Opinion Frames • This blue remote is cool.

• What’s more, the rubbery material is ergonomic.

• We must really go for the blue remote.

• I feel the red remote is a better choice.

• The blue remote will be too expensive.

Sentiment Pos

Sentiment Pos

Arguing Pos

Sentiment Pos

Sentiment NegSPSNaltalternative

same

same

42

Data

• AMI meeting corpus (Carletta et al., 2005)– Rich in opinions– Rich interplay between opinion types, targets,

and polarities– Same and alternative target relations are

prominent– Targets are relatively concrete

43

The Project roadmapAccomplished

– Developed a multi-stage annotation scheme – Tested human reliability in recognizing opinion frames– Showed that the frame presence between two opinion bearing

sentences can be automatically detected– Showed that interdependent interpretation improves automatic

detection of opinion polarityTo Do

– Fully automate effective discourse level opinion interpretation – Text data

• Do the opinion frames behave the same way? What knowledge sources can we use for our task in text?

44

Discourse Level Opinion Graphs (DLOGs)

• Pair-wise relationship from Opinion frames are used to make Discourse Level Opinion Graph (DLOGs)

45

LegendAlternative Target relation ------------Same Target relation ------------Reinforcing frame relation ------------Non-reinforcing frame relation ------------

46

LegendAlternative Target relation ------------Same Target relation ------------Reinforcing frame relation ------------Non-reinforcing frame relation ------------

47

Improving Polarity Classification using DLOG

• Computationally model discourse level relations between opinions

• Leverage these relations to improve polarity classification

48

Improving Polarity Classification using DLOG

• Initial step:– Individual classifiers for recognizing opinions,

polarities, and the 4 types of links• Iterative steps

– Relational classifiers that use information of the neighboring nodes

• The classes assigned to neighbors thus far

49

Illustration• I like the LCD feature

• We must implement the LCD

• I think the LCD is hot

50

DLOG• I like the LCD feature

• We must implement the LCD

• I think the LCD is hot ?

51

DLOG• I like the LCD feature

• We must implement the LCD

• I think the LCD is hot

same

same

52

DLOG• I like the LCD feature

• We must implement the LCD

• I think the LCD is hot

same

same

Reinforcing

53

DLOG• I like the LCD feature

• We must implement the LCD

• I think the LCD is hot

same

same

Reinforcing

Reinforcing

54

DLOG• I like the LCD feature

• We must implement the LCD

• I think the LCD is hot

same

same

Reinforcing

Reinforcing

57

Findings

• Link prediction is hard due to data skewness fully automatic system does not significantly improve polarity classification

58

Findings

• Can discourse-level opinion graphs be exploited to improve the performance of opinion polarity classification?

• Yes!

• 8-16 percentage point improvement in F-measure when the graph structure is known as well as the neighbor’s polarity is known

• 3-5 percentage point improvement in F-measure when only the graph structure is known

59

Upcoming releases

• Reconcile NP coreference system• OpinionFinder version 2

– Improved performance on opinion and polarity recognition

• MPQA opinion annotated corpus version 3– Attitude and target span annotations added – Opinion topic coreference annotations added

• New lexicon representation and toolkit– Uniform representation for different types of subjectivity clues– Attributes, such as polarity, intensity– Toolkit for adding entries to the dictionary and finding instances

of them in a corpus

60

Jodange

• Demo www.jodange.com• Commercialization of methods to identify and

aggregate opinions and their attributes

Opinion FrameExpression: “blasted”Polarity: negative Source: “The Australian Press”Topic: “Italy”

NP Coref ResolutionSource: “The Australian Press” = “their” = “the reporters”

61

Interactions• UACs

– Riloff and Hovy have continued their collaboration on Information Extraction

– Riloff was on sabbatical at ISI through the summer• Other

– Riloff and Cardie are engaged in coreference resolution work with researchers at LLNL

– Wiebe is in the planning stages of work with researchers at LLNL on a news search project

– Riloff is collaborating with the Veterinary Information Network on research in pet health surveillance

– Wiebe is collaborating with researchers in the U. Pittsburgh Medical School on an NIH syndromic surveillance project

– Cardie is collaborating with researchers in the Cornell U. Information Science program on a project on the detection of deception in on-line communication

62

Research Plans

Information extraction for pet health surveillance– collaborating with the Veterinary Information Network

to study message board discussions between veterinary professionals.

– goal is information extraction of entities and relations associated with pet health events, to discover:

• associations between symptoms and diseases• spikes or trends in diseases (e.g., kidney failure

cases during pet food recall)• unusual or unexplained clusters of symptoms or

diseases

63

Research Plans

Coreference resolution– Developing methods to incorporate knowledge from the web as

corroborating evidence for coreference resolution.

• Ex: Orrin Hatch and Ralph Becker are possible antecedents for “the mayor” … issue web queries to decide!

– Experimenting with methods to automatically generate training data for domain-specific coreference resolution.

– Developing state-of-the-art coreference resolver called Reconcile that will be made publicly available.

• testbed for research and analysis

• easily configurable to use different subcomponents

64

Research PlansSelective information extraction with relevant regions– developing weakly supervised learners to create relevant

region classifiers

– learning several different types of clues with varying levels of reliability

• primary indicators are reliable enough to apply everywhere

• secondary indicators are only applied within relevant regions of text

Domains: terrorist events & disease outbreak reports

65

Architecture for Selective IE

UnstructuredText

RelevantSentenceClassifier

IE System

RelevantSentences

IE Patternsand Clues

Extractions

Pattern &Clue Learner

Relevant Region Training

Seed IEPatterns

Relevant andIrrelevant

Documents

66

Research Plans

• Complete the releases of the MPQA corpus, OpinionFinder, & lexicon and toolkit

67

Discourse level opinion interpretation

• Move from task-oriented meetings to blogs and other texts

• Challenges – Other types of relations between targets are more dominant than

in task-oriented meetings– Targets are often more abstract

• Opportunities– Richer lexically– Can exploit parse trees– Explicit discourse cues are more common– Sentences are longer and more self contained

68

Lexicon

• Explore different uses of words, to zero in on the subjective ones

• Example: benefit

69

Lexicon

• Example: benefit• Very often objective, as a Verb: Children with ADHD benefited from a 15-course

of fish oil

70

Lexicon

• Noun uses look more promising: The innovative economic program has shown benefits to humanity

71

Lexicon

• However, there are objective noun uses too: …tax benefits.…employee benefits.…tax benefits to provide a stable economy.…health benefits to cut costs.

72

Lexicon

• Pattern: benefits as the head of a noun phrase containing a prepositional phrase

• Matches this:The innovative economic program has shown

proven benefits to humanity• But none of these:…tax benefits.…employee benefits.…tax benefits to provide a stable economy.…health benefits to cut costs.

73

Entry representationbe soft on crime

<item index="1"><itemMorphoSyntax>

<lemma>be</lemma></itemMorphoSyntax><itemRelation xsi:type="ngramPattern">

<distance>2</distance><landmark>2</landmark></itemRelation></item>

<item index="2"><itemMorphoSyntax>

<word>soft</word><majorClass>J</majorClass></itemMorphoSyntax>

<itemRelation xsi:type="ngramPattern"><distance>1</distance><landmark>3</landmark></itemRelation></item>

<item index="3"><itemMorphoSyntax>

<word>on</word></itemMorphoSyntax><itemRelation xsi:type="ngramPattern">

<distance>1</distance><landmark>4</landmark></itemRelation></item>

<item index="4"><itemMorphoSyntax>

<word>crime</word><majorClass>N</majorClass></itemMorphoSyntax>

74

Attributive information

<entryAttributes origin="j"><name>be soft on crime</name><subjective>true</subjective><reliability>h</reliability><confidence>h</confidence><subType>sen</subType><example>The Obama campaign rejected the notion that the senator

might be vulnerable to accusations that he is soft on crime.</example><morphosyn>vp</morphosyn><target>s</sp_target><polarity>n</polarity><intensity>m</intensity><confidence>h</confidence><regex>1:[morph:[lemma="be"] order:[distance="2" landmark="2"]] 2:

[morph:[word="soft" majorClass="J"] order:[distance="1" landmark="3"]] 3:[morph:[word="on"] order:[distance="1" landmark="4"]] 4:[morph:[word="crime" majorClass="N"]]</regex>

<patterntype>ngramPattern</patterntype>

75

Research Plans

• Learn subjective uses from corpora (bodies of texts)

• Capture longer subjective constructions• Incorporate word senses

– OntoNotes (Hovy)• Add relevant knowledge about expressions• Exploit the richer knowledge resource to improve

opinion extraction