entity/event-level sentiment detection and inference lingjia deng intelligent systems program...
TRANSCRIPT
![Page 1: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/1.jpg)
1
Entity/Event-LevelSentiment Detection and
Inference
Lingjia DengIntelligent Systems Program
University of Pittsburgh
Dr. Janyce Wiebe, Intelligent Systems Program, University of PittsburghDr. Rebecca Hwa, Intelligent Systems Program, University of Pittsburgh
Dr. Yuru Lin, Intelligent Systems Program, University of PittsburghDr. William Cohen, Machine Learning Department, Carnegie Mellon University
![Page 2: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/2.jpg)
3
A World Of Opinions
REVIE
W EDITORIALS
BLOGSTWITTER
NEWS
![Page 3: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/3.jpg)
6
Motivation• ... people protest the country’s same-sex
marriage ban ....
people
positive
negative
protest
same-sex marriage ban
WHAT ABOUT SAME-SEX MARRIAGE?
![Page 4: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/4.jpg)
7
Explicit Opinions• The explicit opinions are revealed by opinion
expressions.
people
positive
negative
protestexplicit
same-sex marriage ban
![Page 5: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/5.jpg)
8
Implicit Opinions• The implicit opinions are not revealed by
expressions, but are indicated in the text.• The system needs to infer implicit opinions.
people
positive
negative
protest
implicit
same-sex marriage ban
![Page 6: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/6.jpg)
9
Goal:Explicit and Implicit Sentiments
people
positive
negative
protest
implicit
explicit
• explicit: negative sentiment• implicit: positive sentiment
same-sex marriage ban
![Page 7: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/7.jpg)
10
Goal:Entity/Event-Level Sentiments• PositivePair(people, same-sex marriage)• NegativePair(people, same-sex marriage ban)
people
positive
negative
protest
implicit
explicit
same-sex marriage ban
![Page 8: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/8.jpg)
11
Three Questions to Solve
• Is there any corpus annotated with both explicit and implicit sentiments?
• No. This proposal develops.
• Is there any inference rules defining how to infer implicit sentiments?
• Yes. (Wiebe and Deng, arXiv, 2014.)
• How do we incorporate the inference rules into computational models?
• This proposal investigates.
![Page 9: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/9.jpg)
12
Completed and Proposed Work
• Expert Annotations on 70 documents (Deng et al., NAACL 2015)
• Non-expert Annotations on hundreds of documents
Corpus:MPQA 3.0
• A corpus of +/-effect event sentiments (Deng et al., ACL 2013)
• A model validating rules (Deng and Wiebe, EACL 2014)
• A model inferring sentiments (Deng et al., COLING 2014)
Sentiment Inference on
+/-Effect Events & Entities
• Joint Models• A pilot study (Deng and Wiebe, EMNLP 2015)
• Extracting Nested Source and Entity/Event Target• Blocking the rules
Sentiment Inference on
General Events & Entities
![Page 10: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/10.jpg)
13
Background:Sentiment Corpora
Genre Source Target ImplicitOpinions
Review SentimentCorpus (Hu and Liu, 2004)
productreviews
writer the product, feature of the product
✗
Sentiment Tree Bank (Socher et al., 2013)
movie reviews
writer the movie✗
MPQA 2.0 (Wiebe at al., 2005; Wilson, 2008)
news, editorials, blogs, etc
writer, and any entity
an arbitrary span
✗
MPQA 3.0 news, editorials, blogs, etc
writer, and any entity
any entity/eventeTarget
✔
(head of noun phrase/verb phrase)
![Page 11: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/11.jpg)
14
Background:MPQA Corpus• Direct subjective
o nested sourceo attitude
• attitude type• target
• Expressive subjective element (ESE)o nested sourceo polarity
• Objective speech evento nested sourceo target
![Page 12: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/12.jpg)
15
MPQA 2.0: An Example
negative attitude
target
nested source:writer, Imam
When the Imam issued the fatwa against
Salman Rushdie for insulting the Prophet…
![Page 13: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/13.jpg)
16
• Explicit sentimentso Extracting explicit opinion expressions, sources and
targets (Wiebe et al., 2005, Johansson and Moschitti, 2013a, Yang and Cardie, 2013, Moilanen and Pulman, 2007, Choi and Cardie, 2008, Moilanen et al., 2010).
• Implicit sentimentso Investigating features directly indicating implicit
sentiment (Zhang and Liu, 2011; Feng et al., 2013). No inference.
o A rules-based system requiring all oracle information. (Wiebe and Deng, arXiv 2014)
Background:Explicit and Implicit Sentiment
![Page 14: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/14.jpg)
18
Completed and Proposed Work
• Expert Annotations on 70 documents (Deng et al., NAACL 2015)
• Non-expert Annotations on hundreds of documents
Corpus:MPQA 3.0
• A corpus of +/-effect event sentiments (Deng et al., ACL 2013)
• A model validating rules (Deng and Wiebe, EACL 2014)
• A model inferring sentiments (Deng et al., COLING 2014)
Sentiment Inference on
+/-Effect Events & Entities
• Joint Models• A pilot study (Deng and Wiebe, EMNLP 2015)
• Extracting Nested Source and Entity/Event Target• Blocking the rules
Sentiment Inference on
General Events & Entities
![Page 15: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/15.jpg)
19
Completed and Proposed Work
• Expert Annotations on 70 documents (Deng et al., NAACL 2015)
• Non-expert Annotations on hundreds of documents
Corpus:MPQA 3.0
• A corpus of +/-effect event sentiments (Deng et al., ACL 2013)
• A model validating rules (Deng and Wiebe, EACL 2014)
• A model inferring sentiments (Deng et al., COLING 2014)
Sentiment Inference on
+/-Effect Events & Entities
• Joint Models• A pilot study (Deng and Wiebe, EMNLP 2015)
• Extracting Nested Source and Entity/Event Target• Blocking the rules
Sentiment Inference on
General Events & Entities
![Page 16: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/16.jpg)
20
From MPQA 2.0 To MPQA 3.0
o “Imam” is negative toward “Rushdie’’.o “Imam” is negative toward “insulting’’.
o “Imam” is NOT negative toward “Prophet”.
negative attitude targ
et
nested source:writer, Imam
When the Imam issued the fatwa against
Salman Rushdie for insulting the Prophet…eTarge
t
![Page 17: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/17.jpg)
21
Expert Annotations• Expert annotators include Dr. Janyce Wiebe and I.
• The expert annotators are asked to select which noun or verb is the eTarget of an attitude or an ESE.
• The expert annotators annotated 70 documents.
• The agreement score is 0.82 on average over four documents.
![Page 18: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/18.jpg)
22
Non-Expert Annotations
• Previous work have tried to ask non expert annotators to annotate subjectivity and opinions (Akkaya et al., 2010, Socher et al., 2013).
• Reliable Annotationso Non-expert annotators with high credits.o Majority vote.o Weighted vote and reliable annotators (Welinder and Perona,
2010).
• Validating Annotation Schemeo 70 documents: Compare non-expert annotations with expert
annotations.o Then, collect non-expert annotations for the remaining corpus.
![Page 19: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/19.jpg)
27
Part 1 Summary
• An entity/event-level sentiment corpus, MPQA 3.0
• Complete expert annotationso 70 documents (Deng and Wiebe, NAACL 2015).
• Propose non-expert annotationso Remaining hundreds of documents.o Crowdsourcing tasks.o Automatically acquiring reliable labels.
![Page 20: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/20.jpg)
28
Completed and Proposed Work
• Expert Annotations on 70 documents (Deng et al., NAACL 2015)
• Non-expert Annotations on hundreds of documents
Corpus:MPQA 3.0
• A corpus of +/-effect event sentiments (Deng et al., ACL 2013)
• A model validating rules (Deng and Wiebe, EACL 2014)
• A model inferring sentiments (Deng et al., COLING 2014)
Sentiment Inference on
+/-Effect Events & Entities
• Joint Models• A pilot study (Deng et al., EMNLP 2015)
• Extracting Nested Source and Entity/Event Target• Blocking the rules
Sentiment Inference on
General Events & Entities
![Page 21: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/21.jpg)
29
+/-Effect Event Definition
• A +effect event has benefiting effect on the theme.o help, increase, etc
• A –effect event has harmful effect on the theme.o harm, decrease, etc
• A triple• <agent, event, theme>
He rejects the paper.
-effect event: rejectAgent: Hetheme: paper
<He, reject, paper>
![Page 22: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/22.jpg)
31
• +Effect(x)o x is a +effect event
• -Effect(x)o x is a –effect event
• Agent(x,a)o a is the agent of +/-effect event x
• Theme(x, h)o h is the theme of +/-effect event x
+/-Effect Event Representation
![Page 23: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/23.jpg)
32
+/-Effect Event Corpus• +/-Effect event information is annotated.
o The +/-effect events.o The agents.o The themes.
• The writer’s sentiments toward the agents and themes are annotated.o positive, negative, neutral
• 134 political editorials.
![Page 24: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/24.jpg)
33
Sentiment Inference Rules
• people protest the country’s same-sex marriage ban.
• explicit sentimento NegativePair(people, ban)
• +/-effect event informationo -Effect(ban)o Theme(ban, same-sex marriage)
NegativePair(people, ban) ^ -Effect(ban) ^ Theme(ban, same-sex marriage)
PositivePair(people, same-sex marriage)
![Page 25: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/25.jpg)
34
• +Effect Rule:• If two entities participate in a +effect event,• the writer’s sentiments toward the entities are
the same.
• -Effect Rule:• If two entities participate in a –effect event,• the writer’s sentiments toward the entities are
the opposite.
Sentiment Inference Rules
Can rules infer sentiments
correctly?(Deng and Wiebe, EACL
2014)
![Page 26: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/26.jpg)
35
Building the graph from
annotations
A
E
D C
B
agent/
theme
• node score: two sentiment scores
(Deng and Wiebe, EACL 2014)
![Page 27: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/27.jpg)
36
Building the graph from
annotations
• edge score: four sentiment constraints scores
• • the score that the sentiment toward D is positive• AND the sentiment toward E is positive
+/-effect
A
E
D C
B
![Page 28: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/28.jpg)
37
Building the graph from
annotations
• edge score: inference rules• if +effect:• if –effect:
+/-effect
A
E
D C
B
![Page 29: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/29.jpg)
38
Loopy Belief Propagation
• Input: the gold standard sentiment of one node• Model: Loopy Belief Propagation• Output: the propagated sentiments of other nodes
+/-effect
A
E
D C
B
agent/
theme
![Page 30: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/30.jpg)
39
Propagating sentiments
• For node E, • can it be propagated with correct sentiment
labels?
+/-effect
A
E
D C
B
agent/
theme
![Page 31: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/31.jpg)
40
Propagating sentiments A E
• Node A is assigned with gold standard sentiment.• Run the propagation.• Record whether Node E is propagated correctly or
not.
+/-effect
A
E
D C
B
agent/
theme
![Page 32: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/32.jpg)
41
Propagating sentiments B E
• Node B is assigned with gold standard sentiment.• Run the propagation.• Record whether Node E is propagated correctly or
not.
+/-effect
A
E
D C
B
agent/
theme
![Page 33: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/33.jpg)
42
Propagating sentiments C E
• Node C is assigned with gold standard sentiment.• Run the propagation.• Record whether Node E is propagated correctly or
not.
+/-effect
A
E
D C
B
agent/
theme
![Page 34: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/34.jpg)
43
Propagating sentiments D E
• Node D is assigned with gold standard sentiment.• Run the propagation.• Record whether Node E is propagated correctly or
not.
+/-effect
A
E
D C
B
agent/
theme
![Page 35: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/35.jpg)
44
Evaluating E being propagated
correctly
• Node E is propagated with sentiment 4 times.• correctness =
(# node E being propagated correctly)/ 4
• average correctness = 88.74%
+/-effect
A
E
D C
B
agent/
theme
![Page 36: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/36.jpg)
45
Conclusion
• Defining the graph-based model with sentiment inference rules.
• Propagating sentiments correctly in 88.74% cases.
• To validate the inference rules only,• The graph-based propagation model is built from
manual annotations.Can we automatically infer
sentiments?(Deng et al., COLING 2014)
![Page 37: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/37.jpg)
46
Local Detectors
Agent1 Agent2 reversed +effect -effect Theme1 Theme2
pos: 0.7neg: 0.5
pos: 0.5neg: 0.6
pos: 0.5neg: 0.5
pos: 0.7neg: 0.5
reverser: 0.9
+effect: 0.8 -effect: 0.2
(Q1) is it +effect or -effect?
(Q2) is the effect reversed?
(Q3) which spans are agents and themes?
(Q4) what are the writer’s sentiments?
• Given a +/-effect event span in a document,• Run state-of-the-art systems assigning local scores.
(Deng et al., COLING 2014)
![Page 38: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/38.jpg)
47
Local Detectors
Agent1 Agent2 reversed +effect -effect Theme1 Theme2
pos: 0.7neg: 0.5
pos: 0.5neg: 0.6
pos: 0.5neg: 0.5
pos: 0.7neg: 0.5
reverser: 0.9
+effect: 0.8 -effect: 0.2
(Q1) word sense disambiguation
(Q2) negation detected
(Q3) semantic role labeling (Q4) sentiment
analysis
(Deng et al., COLING 2014)
Ambiguity
![Page 39: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/39.jpg)
48
Global Optimization
• The global model selects an optimal set of candidates:o one candidate from the four agent sentiment candidates,
• Agent1-pos, Agent1-neg, Agent2-pos, Agent2-nego one/no candidate from the reversed candidate,o one candidate from the +/-effect candidates,o one candidate from the four theme sentiment candidates.
pos: 0.7neg: 0.5
pos: 0.5neg: 0.6
pos: 0.5neg: 0.5
pos: 0.7neg: 0.5
reverser: 0.9
+effect: 0.8 -effect: 0.2
Agent1 Agent2 reversed +effect -effect Theme1 Theme2
![Page 40: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/40.jpg)
49
Objective Function
• The framework assigns values (0 or 1) to uo maximizing the scores given by the local
detectors,
• and assigns values (0 or 1) to ξ, δo minimizing the cases where +/-effect event
sentiment rules are violated.
• Integer Linear Programming (ILP) is used.
p: candidate local score
u: binary indicator of choosing candidate
ξ, δ: slack variables of triple <i,k,j>representing this triple is an exception to +effect –effect rule (exception: 1)
![Page 41: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/41.jpg)
50
• In a +effect event, sentiments are the same
+Effect Rule Constraints
1 001 10 0 +effect:
1-effect: 0
exception: 1not exception: 0
AND
![Page 42: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/42.jpg)
51
-Effect Rule Constraints
• In a –effect event, sentiments are opposite.
![Page 43: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/43.jpg)
52
Performances
Accur
acy
of Q
1
Accur
acy
of Q
2
Accur
acy
of Q
3
F-mea
sure
of Q
40
0.4
0.8
Light Color: LocalDark Color: ILP
(Q1) is it +effect or -effect?
(Q2) is the effect reversed?
(Q3) which spans are agents and themes?
(Q4) what are the writer’s sentiments?
Precision Q4
Recall of Q4
![Page 44: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/44.jpg)
56
Part 2 Summary• Inferring sentiments toward entities participating
in the +/-effect events.
• Developed an annotated corpus (Deng et al., ACL 2013).
• Developed a graph-based propagation model showing the inference ability of rules (Deng and Wiebe, EACL 2014).
• Developed an Integer Linear Programming model jointly resolving various ambiguities w.r.t. +/-effect events and sentiments (Deng at al., COLING 2014).
![Page 45: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/45.jpg)
57
Completed and Proposed Work
• Expert Annotations on 70 documents (Deng et al., NAACL 2015)
• Non-expert Annotations on hundreds of documents
Corpus:MPQA 3.0
• A corpus of +/-effect event sentiments (Deng et al., ACL 2013)
• A model validating rules (Deng and Wiebe, EACL 2014)
• A model inferring sentiments (Deng et al., COLING 2014)
Sentiment Inference on
+/-Effect Events & Entities
• Joint Models• A pilot study (Deng and Wiebe, EMNLP 2015)
• Extracting Nested Source and Entity/Event Target• Blocking the rules
Sentiment Inference on
General Events & Entities
![Page 46: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/46.jpg)
59
Joint Models• In (Deng et al., COLING 2014), we use Integer
Linear Programming framework.
• Local systems are run.• Joint models take local scores as input, and
sentiment inference rules as constraints.
• In ILP, the rules are written in equations and in equations.
![Page 47: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/47.jpg)
60
Joint Models: General Inference Rules• Great! Dr. Thompson likes the project. …
• Explicit sentiment: o Positive(Great)o Source(Great, speaker)o ETarget(Great, likes)o PositivePair(speaker, likes)
• Explicit sentiment:o Positive(likes)o Source(likes, Dr. Thompson)o ETarget(likes, project)o PositivePair(Dr. Thompson, project)
![Page 48: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/48.jpg)
61
Joint Models: General Inference Rules• Great! Dr. Thompson likes the project. …
• Explicit sentiment: o Positive(Great)o Source(Great, speaker)o ETarget(Great, likes)o PositivePair(speaker, likes)
• Explicit sentiment:o Positive(likes)o Source(likes, Dr. Thompson)o ETarget(likes, project)o PositivePair(Dr. Thompson, project)
PositivePair(speaker, likes) ^ Positive(likes) ^ ETarget(likes, project) PositivePair(spkear, project)
sentimenttoward
sentiment
![Page 49: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/49.jpg)
62
Joint Models• More complex rules, in first order logics.
• Markov Logic Network (Richardson and Domingos, 2006).o a set of atoms to be groundedo a set of weighted if-then rules
o rule: friend(a,b) ^ voteFor(a,c) voteFor(b,c)o atom: friend(a,b), voteFor(a,c)o ground atom: friend (Mary, Tom)
• MLN selects a set of ground atoms that maximize the number of satisfied rules.
![Page 50: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/50.jpg)
63
• PositivePair(s,t)• NegativePair(s,t)
• Positive(y) Negative(y)• Source(y,s) Etarget(y,t)
• +Effect(x) -Effect(y)• Agent(x,a) Theme(x,a)
Joint Model:Pilot Study Atoms
Predicted by joint models
Assigned scores bylocal systems
![Page 51: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/51.jpg)
66
• 3 joint models:
• Joint-1 (without any inference)• Joint-2 (added general sentiment inference rules)• Joint-3 (added +/-effect event information and the
rules)
Joint Model:Pilot Study Experiments
![Page 52: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/52.jpg)
69
• Task: extracting PostivePair(s,t) and NegativePair(s,t).
• Baselines: for an opinion extracted by state-of-the-art systemso source s: the head of extracted source span
o eTarget t:• ALL NP/VP:
o all the nouns and verbs are eTargets• Opinion/Target Span Heads (state-of-the-art):
o head of extracted target span; o head of opinion span
o PositivePair or NegativePair: the extracted polarity
Joint Model:Pilot Study Performances
![Page 53: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/53.jpg)
70
• Task: extracting PostivePairs and NegativePairs.
Joint Model:Pilot Study Performances
PosP
air A
ccur
acy
Neg
Pair
Accur
acy
PosP
air F
-mea
sure
Neg
Pair
F-mea
sure
0
0.1
0.2
0.3
0.4
0.5
ALL NP/VPOpinion/Target Span HeadsPSL1PSL2PSL3
True Negatives
![Page 54: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/54.jpg)
71
• We cannot directly use the state-of-the-art sentiment analysis system outputs (spans) for entity/event-level sentiment analysis task.
• The inference rules can find more entity/event-level sentiments.
• The most basic joint models in our pilot study can improve in accuracies.
Joint Model:Pilot Study Conclusions
![Page 55: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/55.jpg)
72
• Various variations of Markov Logic Network.• Integer Linear Programming.
• Each local component being improved.o Nested Sourceso ETargeto Blocking the rules
Joint Model:Proposed Extensions
![Page 56: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/56.jpg)
74
Nested Sources nested source:writer, Imam,
Rushdie
negative attitude
When the Imam issued the fatwa against Salman Rushdie for insulting the Prophet…
• How do we know Rushdie is negative toward Prophet?• Because Imam claims so, by issuing the fatwa against
him.
• How do we know Imam has issued the fatwa?• Because the writer tells us so.
![Page 57: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/57.jpg)
75
Nested Sourcesnested source:writer, Imam,
Rushdie
negative attitude
When the Imam issued the fatwa against Salman Rushdie for insulting the Prophet…
• Nested source reveals the embedded private states in MPQA.
• Attributing quotations (Pareti et al., 2013, de La Clergerie et al., 2011, Almeida et al., 2014).
• The overlapped opinions and opinion targets.
![Page 58: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/58.jpg)
76
• Extracting named entities and events as potential eTargets (Pan et al., 2015, Finkel et al., 2005, Nadeau and Sekine, 2007, Li et al., 2013, Chen et al., 2009, Chen and Ji, 2009).
• Entity co-reference resolution (Haghighi and Klein, 2009;
Haghighi and Klein, 2010; Song et al., 2012).• Event co-reference resolution (Li et al., 2013, Chen et
al., 2009, Chen and Ji, 2009).
• Integrating external world knowledgeo Entity Linking to Wikipedia (Ji and Grishman, 2011; Milne and
Witten, 2008; Dai et al., 2011; Rao et al., 2013)
ETarget (Entity/Event Target)
![Page 59: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/59.jpg)
79
• That man killed the lovely squirrel on purpose.o Positive toward squirrelo killing is a –effect evento Negative toward that man
• That man accidentally hurt the lovely squirrel.o Positive toward the squirrelo hurting is a –effect evento Negative toward that man
Blocking Inference Rules
✗
✓
![Page 60: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/60.jpg)
80
• That man killed the lovely squirrel on purpose.o Positive toward squirrelo killing is a –effect evento Negative toward that man
• That man accidentally hurt the lovely squirrel.o Positive toward the squirrelo hurting is a –effect evento Negative toward that man
Blocking Inference Rules
✗
✓
![Page 61: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/61.jpg)
81
Collect Blocking
Cases
Compare and Find differenc
es
Learn to Recogniz
e
Blocking Inference Rules
![Page 62: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/62.jpg)
84
Part 3 Summary• Joint models:
o A pilot study (Deng and Wiebe, EMNLP 2015)o Improved joint models integrating improved
components.
• Explicit Opinions:o Opinion expressions and polarities (state-of-the-art)o Opinion nested sourceso Opinion eTargets (entity/event-level targets)
• Implicit Opinions:o General inference rules (Wiebe and Deng, arxIV 2014)o When the rules are blocked
![Page 63: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/63.jpg)
85
Completed and Proposed Work
• Expert Annotations on 70 documents (Deng et al., NAACL 2015)
• Non-expert Annotations on hundreds of documents
Corpus:MPQA 3.0
• A corpus of +/-effect event sentiments (Deng et al., ACL 2013)
• A model validating rules (Deng and Wiebe, EACL 2014)
• A model inferring sentiments (Deng et al., COLING 2014)
Sentiment Inference on
+/-Effect Events & Entities
• Joint Models• A pilot study (Deng and Wiebe, EMNLP 2015)
• Extracting Nested Source and Entity/Event Target• Blocking the rules
Sentiment Inference on
General Events & Entities
![Page 64: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/64.jpg)
86
Research Statement• Defining a new sentiment analysis task
(entity/event-level sentiment analysis task), • this work develops annotated corpora as
resources of the task • and investigates joint prediction models • integrating explicit sentiments, entity or event
information and inference rules together • to automatically recognize both explicit and
implicit sentiments expressed among entities and events in the text.
sentiment analysis
information extraction
![Page 65: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/65.jpg)
87
Timeline Date Content Deliverable results
Sep - Nov Collecting Non-Expert Annotation Completed MPQA 3.0 corpus
Nov - Jan Extracting Nested Source and ETarget
NAACL 2016: A System Extracting Nested Source & Entity/Event-Level ETarget
Jan - Mar Analyzing Blocked RulesACL/COLING/EMNLP 2016:Improved Graph-Based Model Performances
Mar - May
An Improved Joint Model Integrating improved Components
Journals submitted
May - Aug Thesis Writing Thesis Ready for Defense
Aug - Dec Thesis Revising Completed Thesis
![Page 66: Entity/Event-Level Sentiment Detection and Inference Lingjia Deng Intelligent Systems Program University of Pittsburgh 1 Dr. Janyce Wiebe, Intelligent](https://reader035.vdocuments.mx/reader035/viewer/2022062422/56649ed85503460f94be6a47/html5/thumbnails/66.jpg)
88