annotating opinions in the world press

Annotating Opinions in the Annotating Opinions in the World PressWorld Press

Theresa Wilson and Janyce WiebeTheresa Wilson and Janyce WiebeUniversity of PittsburghUniversity of PittsburghIntelligent Systems ProgramIntelligent Systems Program

andandDepartment of Computer ScienceDepartment of Computer Science

July 5, 2003July 5, 2003 SIGdial 2003SIGdial 2003 22

IntroductionIntroduction

Private State: mental or emotional state, which cannot be directly observed or verified (Quirk et al., 1985)

EmotionsSpeculation

BeliefsOpinions

Evaluations


Private States in TextPrivate States in TextSubjectivity: aspects of language used to express

private states, including opinions, emotions, evaluations, speculations, sentiments (Banfield 1982)

“Complained”“You Idiot!”

“Terrible product”

“Speculated”“Maybe”

“I wonder”

“Enthused”“Wonderful!”“Great idea”

(See Fodor 1979, Cohn 1978, Chafe 1986, Fludernik 1993, Wiebe 1994, Stein & Wright 1995 for additional references.)


Why do we care?Why do we care?

News articles mix of News articles mix of factual/subjective languagefactual/subjective language

Help existing NLP applications: IE, IR, Help existing NLP applications: IE, IR, summarization systems, etc.summarization systems, etc.

Support new tasks: multi-perspective Support new tasks: multi-perspective question answeringquestion answering


Direct mentions of private statesDirect mentions of private statesThe United States The United States fearsfears a spill-over from the a spill-over from the anti-terrorist campaign.anti-terrorist campaign.

Expressive subjective elementsExpressive subjective elementsThe part of the US human rights report about The part of the US human rights report about China is China is full of absurdities and fabricationsfull of absurdities and fabrications..

Two Main Ways of Expressing Two Main Ways of Expressing Private StatesPrivate States


Private States and Speech Private States and Speech EventsEvents

Speech Event Speech Event speaking or writing speaking or writing

““We foresaw electoral fraud but not daylight We foresaw electoral fraud but not daylight robbery,” Tsvangirai robbery,” Tsvangirai saidsaid..

Medical Department head Dr. Hameed Saeed Medical Department head Dr. Hameed Saeed saidsaid the patient’s blood has been sent to the Institute for the patient’s blood has been sent to the Institute for Virology in Johannesburg for analysis.Virology in Johannesburg for analysis.


Nested SourcesNested Sources

“The US fears a spill-over’’, said Xirao-Nima, a professorof foreign affairs at the Central University for Nationalities.

(writer, Xirao-Nima, US) (writer, Xirao-Nima)(writer)

“The report is full of absurdities,’’ he continued.

(writer, Xirao-Nima) (writer, Xirao-Nima)(writer)


Annotation SchemeAnnotation Scheme

Implemented in GATE Implemented in GATE ((http://http://gate.ac.ukgate.ac.uk))

Shorthand for direct private state or speech

event expression

http://gate.ac.uk/

http://gate.ac.uk/


Attributes for Annotation Attributes for Annotation TypesTypes

On (Direct Private On (Direct Private StateState

or Speech Event)or Speech Event) nested-sourcenested-source onlyfactiveonlyfactive overall-strengthoverall-strength on-strengthon-strength attitude-typeattitude-type attitude-towardattitude-toward is-implicitis-implicit minorminor

Expressive SubjectiveExpressive SubjectiveElementElement

nested-sourcenested-source strengthstrength attitude-typeattitude-type



On (Direct Private StateOn (Direct Private Stateor Speech Event)or Speech Event)

nested-sourcenested-source onlyfactiveonlyfactive overall-strengthoverall-strength on-strengthon-strength attitude-typeattitude-type attitude-towardattitude-toward is-implicitis-implicit minorminor




onlyfactive Attributeonlyfactive Attribute

“The US fears a spill-over’’, said Xirao-Nima, a professor of foreign affairs at the Central University for Nationalities.

(writer)onlyfactive=yes

(writer, Xirao-Nima)onlyfactive=yes

(writer, Xirao-Nima, US)onlyfactive=no


Important! onlyfactive=no Important! onlyfactive=no may still contain factsmay still contain facts

““John John criticizedcriticized Mary for smoking.” Mary for smoking.”

(writer, John)onlyfactive=no


The Foreign Ministry said . . .

on-strength=neutral

The Foreign Ministry objected . . .

The Foreign Ministry vehemently denied . . .

on-strength=medium

Strength of Private StateStrength of Private State

on-strength=high

. . . inaccurate section on Chechnya.

strength=low

. . . odious section on Chechnya.

strength=high


Overall Strength of Private StateOverall Strength of Private State

overall-strength=high

The Foreign Ministry objected in particular to the “odious” section on Chechnya.

on-strength=medium

strength=high

“We foresaw electoral fraud but not daylight robbery,” Tsvangirai said.

strength=medium

strength=medium

on-strength=neutraloverall-

strength=medium


DataData

English-language versions of articles English-language versions of articles from world press (FBIS + few other from world press (FBIS + few other sources)sources)

8500 sentences have been annotated8500 sentences have been annotated Agreement StudyAgreement Study

3 annotators (A, M, S)3 annotators (A, M, S) 210 sentences (13 documents)210 sentences (13 documents) 1/3 news, 1/3 news on “hot-topic”, 1/3 1/3 news, 1/3 news on “hot-topic”, 1/3

editorialseditorials


Types of AgreementTypes of Agreement

Agreement for spansAgreement for spans OnsOns Expressive Subjective ElementsExpressive Subjective Elements

Agreement for ON Agreement for ON onlyfactive onlyfactive attributeattribute

Agreement for sentencesAgreement for sentences Derive sentence-level annotations from Derive sentence-level annotations from

lower-level annotationslower-level annotations


Agreement Issues: SpansAgreement Issues: Spans

Overlapping spans ok Overlapping spans ok it’s a match it’s a match

A: His comments were immediately contested . . .

M: His comments were immediately contested . . .


Agreement Issues: SpansAgreement Issues: Spans

A: Russian military policy has been to keep a distance from . . .

M: Russian military policy has been to keep a distance from . . .

(writer, Russian Miliatary)

(writer, Russian Miliatary)

Different set of annotations for each Different set of annotations for each annotator annotator Kappa inappropriate Kappa inappropriate


Annotator A

Annotator B

Agreement Evaluation: Agreement Evaluation: spansspans

Agreement Agreement measured by measured by percentage of A’s percentage of A’s constituents constituents matching B’s, and matching B’s, and vice versavice versa

agr (a||b) = |A matching B| |A|


Span Agreement: OnsSpan Agreement: Ons

aa bb agr(a||b)agr(a||b) agr(b||a)agr(b||a) averagaveragee

AA MM 0.750.75 0.910.91

AA SS 0.800.80 0.850.85

MM SS 0.860.86 0.750.75

0.820.82


Measuring Agreement: Measuring Agreement: ON ON onlyfactive onlyfactive

A:A: Reformers Reformers toldtold Putin all along that he should take Putin all along that he should take advantage of "the rare opportunity for (Russia) to become advantage of "the rare opportunity for (Russia) to become integrated with the civilized world." integrated with the civilized world." But main stream But main stream Russian military Russian military policypolicy has been to keep a distance from has been to keep a distance from America.America.M:M: Reformers Reformers toldtold Putin all along that he should take Putin all along that he should take advantage of "the rare opportunity for (Russia) to become advantage of "the rare opportunity for (Russia) to become integrated with the civilized world.“ integrated with the civilized world.“ But main stream But main stream Russian military policy has been to Russian military policy has been to keep a distancekeep a distance from from America.America. writer on



A:A: Reformers Reformers toldtold Putin all along that he should take Putin all along that he should take advantage of "the rare opportunity for (Russia) to become advantage of "the rare opportunity for (Russia) to become integrated with the civilized world." integrated with the civilized world." But main stream But main stream Russian military Russian military policypolicy has been to keep a distance from has been to keep a distance from America.America.M:M: Reformers Reformers toldtold Putin all along that he should take Putin all along that he should take advantage of "the rare opportunity for (Russia) to become advantage of "the rare opportunity for (Russia) to become integrated with the civilized world.“ integrated with the civilized world.“ But main stream But main stream Russian military policy has been to Russian military policy has been to keep a distancekeep a distance from from America.America. writer on



A: A: Reformers Reformers toldtold Putin all along that he should take Putin all along that he should take advantage of "the rare opportunity for (Russia) to become advantage of "the rare opportunity for (Russia) to become integrated with the civilized world." integrated with the civilized world." But main stream But main stream Russian military Russian military policypolicy has been to keep a distance from has been to keep a distance from America.America.M: M: Reformers Reformers toldtold Putin all along that he should take Putin all along that he should take advantage of "the rare opportunity for (Russia) to become advantage of "the rare opportunity for (Russia) to become integrated with the civilized world.“ integrated with the civilized world.“ But main stream But main stream Russian military policy has been to Russian military policy has been to keep a distancekeep a distance from from America.America. writer on

onlyfactive=yes

onlyfactive=no

onlyfactive=yes

onlyfactive=no

onlyfactive=no

onlyfactive=yes


Agreement results: Agreement results: onlyfactiveonlyfactive

0.81

0.89

0.7

0.75

0.8

0.85

0.9

borderline-onlyfactiveremoved (10% )

BorderlineBorderline onlyfactive:onlyfactive:1.1. At least one annotator At least one annotator

marked expression marked expression onlyfactive=noonlyfactive=no

2.2. Neither annotator Neither annotator marked expression marked expression overall-strengthoverall-strength > low > low

Ave Kappa


Deriving Sentence-level Deriving Sentence-level AnnotationsAnnotations

Sue Sue saidsaid, “The election was Tuesday.” , “The election was Tuesday.”


(writer, Sue)onlyfactive=yes Objective

Sue Sue saidsaid, “The election was , “The election was fairfair.” .”


(writer, Sue)onlyfactive=no Subjective


Agreement results: Agreement results: sentence-level subjective sentence-level subjective

annotationsannotations

0.77

0.87

0.7

0.75

0.8

0.85

0.9

borderline subjectivesentences removed (11% )

Borderline subjectiveBorderline subjective sentence:sentence:

1.1. At least one At least one annotator marked at annotator marked at least one expression least one expression onlyfactive=noonlyfactive=no

2.2. Neither annotator Neither annotator marked an marked an overallstrengthoverallstrength > > lowlow

Ave Kappa


ConclusionsConclusions

Detailed scheme for annotation of opinions Detailed scheme for annotation of opinions and other private statesand other private states Good agreement between annotatorsGood agreement between annotators

Strong agreement about clear cases of Strong agreement about clear cases of subjectivitysubjectivity

Lower-level annotations Lower-level annotations higher-level higher-level subjective sentence annotationssubjective sentence annotations

ExtendableExtendable


Using GATE to perform Using GATE to perform private state annotationsprivate state annotations

GATE GATE http://gate.ac.ukhttp://gate.ac.uk

General annotation instructions and General annotation instructions and GATE instructionsGATE instructionshttp://www.cs.pitt.edu/~wiebe/pubs/ardasummerhttp://www.cs.pitt.edu/~wiebe/pubs/ardasummer0202

http://gate.ac.uk/

http://www.cs.pitt.edu/~wiebe/pubs/ardasummer02

http://www.cs.pitt.edu/~wiebe/pubs/ardasummer02


Thank youThank you

This work was performed in support of the Northeast This work was performed in support of the Northeast Regional Research Center (NRRC) which is sponsored by the Regional Research Center (NRRC) which is sponsored by the Advanced Research and Development Activity in Information Advanced Research and Development Activity in Information Technology (ARDA), a U.S. Government entity which Technology (ARDA), a U.S. Government entity which sponsors and promotes research of import to the Intelligence sponsors and promotes research of import to the Intelligence Community which includes but is not limited to the CIA, DIA, Community which includes but is not limited to the CIA, DIA, NSA, NIMA and NRO.NSA, NIMA and NRO.


Contingency Table: Contingency Table: onlyfactive judgementsonlyfactive judgements

Tagger MTagger M

YesYes NoNo

TaggeTaggerr

AA

YeYess

nnyyyy=18=1811

nnynyn=25=25

NoNo nnnyny=12=12 nnnnnn=25=2522Agreement for

onlyfactive judgments

Tagger MTagger M

YesYes NoNo

TaggeTaggerr

AA

YeYess

nnyyyy=18=1811

nnynyn=8=8

NoNo nnnyny=11=11 nnnnnn=22=2244Agreement for

onlyfactive judgments, borderline cases

removed


Onlyfactive AgreementOnlyfactive Agreement

All All ExpressionsExpressions

Borderline RemovedBorderline Removed

KappaKappa agreeagree KappaKappa agreeagree % removed% removed

A & MA & M 0.840.84 0.910.91 0.940.94 0.960.96 1010

A & SA & S 0.840.84 0.920.92 0.900.90 0.950.95 88

M & SM & S 0.740.74 0.870.87 0.840.84 0.920.92 1212


Subjective Sentence Subjective Sentence AgreementAgreement

All All ExpressionsExpressions

Borderline RemovedBorderline Removed

KappaKappa agreeagree KappaKappa agreeagree % removed% removed

A & MA & M 0.750.75 0.890.89 0.870.87 0.950.95 1111

A & SA & S 0.840.84 0.940.94 0.920.92 0.970.97 88

M & SM & S 0.720.72 0.880.88 0.830.83 0.930.93 1313


Why we exclude minor on Why we exclude minor on annotationsannotations

This is the first time that the Cuban government has expressed its position over the U.S. holding of the Taliban and al-Qaeda prisoners in its territory.

(writer, Cuban gov)onlyfactive=no

Objective Sentence!

annotating opinions in the world press

Documents