joint models of disagreement and stance in online debate dhanya sridhar, james foulds, bert huang,...
Post on 02-Jan-2016
230 Views
Preview:
TRANSCRIPT
1
Joint Models of Disagreement and Stance in Online Debate
Dhanya Sridhar, James Foulds, Bert Huang, Lise Getoor, Marilyn Walker
University of California, Santa CruzVirginia Institute of Technology
1 21
1 1
1
2
2
• Social media sites for debating issues
• Valuable resources for:– Argumentation– Dialogue– Sentiment– Opinion mining
Online Debate Forums
3
CreateDebate.org
4
CreateDebate.org Debate topic
5
CreateDebate.org Debate topic
Posts
6
CreateDebate.org Debate topic
Posts
Replies
7
CreateDebate.org Debate topic
Posts
Replies
Reply polarity
8
4Forums.com
9
4Forums.comQuotation
10
Graph of posts:tree structure
Online Debate Forums
Graph of users:loopy structure= reply link
11
Graph of posts:tree structure
Online Debate Forums
Graph of users:loopy structure
12
Zoom in on an Example
13
Zoom in on an Example
14
I believe Obama is likely
the worst!
He’s been infinitely more effective than
Bush!
Zoom in on an Example
15
I believe Obama is likely
the worst!
He’s been infinitely more effective than
Bush!
Anti Obama
Pro Obama
Task: Stance Classification
Useful for advocacy and get-out-the-vote campaigns
16
I believe Obama is likely
the worst!
He’s been infinitely more effective than
Bush!
Anti Obama
Pro Obama
Task: Stance Classification
Study argumentation and dialogue
Posts express stance
17
I believe Obama is likely
the worst!
He’s been infinitely more effective than
Bush!
Anti Obama
Pro Obama
Task: Disagreement Classification
Study argumentation and dialogue
Disagree on stance
18
I believe Obama is likely
the worst!
He’s been infinitely more effective than
Bush!
Anti Obama
Pro Obama
Task: Disagreement Classification
Study argumentation and dialogue
Disagree on stance
Posts express disagreement
19
Stance
Stance
Stance
Stance
Classification Targets
• Stance• Author-level• Post-level
• Disagreement• Author-level• Post-level• Textual
20
Stance
Stance
Stance
Stance
Disagrees
Disagrees
Disagrees
Disagrees
Classification Targets
• Stance• Author-level• Post-level
• Disagreement• Author-level• Post-level• Textual
21
Stance
Stance
Stance
Stance
Disagrees
Disagrees
Disagrees
Disagrees
Classification Targets
• Stance• Author-level• Post-level
• Disagreement• Author-level• Post-level• Textual
22
Related Work• Walker et. al (2012), Decision Support Sciences
State-of-the-art local stance classifier using linguistic features
• Walker et. al (2012), NAACLCollective stance classification using MaxCut over rebuttal links
• Hasan and Ng. (2013), EMNLPCollective stance sequence labeling with CRF; benefits of user stance consistency constraint
• Abbott et. al (2012), ACLLocal disagreement classifier using linguistic features
• Burfoot et. al (2011), ACLJoint classification of stance and reply polarity in Congressional debates
23
Related Work• Walker et. al (2012), Decision Support Sciences
State-of-the-art local stance classifier using linguistic features
• Walker et. al (2012), NAACLCollective stance classification using MaxCut over rebuttal links
• Hasan and Ng. (2013), EMNLPCollective stance sequence labeling with CRF; benefits of user stance consistency constraint
• Abbott et. al (2012), ACLLocal disagreement classifier using linguistic features
• Burfoot et. al (2011), ACLJoint classification of stance and reply polarity in Congressional debates
24
Related Work• Walker et. al (2012), Decision Support Sciences
State-of-the-art local stance classifier using linguistic features
• Walker et. al (2012), NAACLCollective stance classification using MaxCut over rebuttal links
• Hasan and Ng. (2013), EMNLPCollective stance sequence labeling with CRF; benefits of user stance consistency constraint
• Abbott et. al (2012), ACLLocal disagreement classifier using linguistic features
• Burfoot et. al (2011), ACLJoint classification of stance and reply polarity in Congressional debates
25
Related Work• Walker et. al (2012), Decision Support Sciences
State-of-the-art local stance classifier using linguistic features
• Walker et. al (2012), NAACLCollective stance classification using MaxCut over rebuttal links
• Hasan and Ng. (2013), EMNLPCollective stance sequence labeling with CRF; benefits of user stance consistency constraint
• Abbott et. al (2012), ACLLocal disagreement classifier using linguistic features
• Burfoot et. al (2011), ACLJoint classification of stance and reply polarity in Congressional debates
26
Related Work• Walker et. al (2012), Decision Support Sciences
State-of-the-art local stance classifier using linguistic features
• Walker et. al (2012), NAACLCollective stance classification using MaxCut over rebuttal links
• Hasan and Ng. (2013), EMNLPCollective stance sequence labeling with CRF; benefits of user stance consistency constraint
• Abbott et. al (2012), ACLLocal disagreement classifier using linguistic features
• Burfoot et. al (2011), ACLJoint classification of stance and reply polarity in Congressional debates
27
Stance Classification:“Teach the Controversy”
• Previous work employs many modeling strategies
• How best to model stance in online debate?
• Answers may be different to Congressional debates– Links have different semantics– Posts much shorter than speeches. Many posts per author– Dialogue is informal
28
Stance Classification:“Teach the Controversy”
• Previous work employs many modeling strategies
• How best to model stance in online debate?
• Answers may be different to Congressional debates– Links have different semantics– Posts much shorter than speeches. Many posts per author– Dialogue is informal
29
Stance
Stance
Stance
Stance
Stance
Stance
Stance
Stance
Stance
Modeling at author-level or post-level?
[Hasan and Ng 2013] [Other Related Work]
Modeling Question 1)
30
Stance
Stance
Stance
Stance
Stance
Stance
Stance
Stance
Stance
Modeling at author-level or post-level?
[Hasan and Ng 2013] [Other Related Work]
Modeling Question 1)
31
Stance
Stance
Stance
Stance
Stance
Stance
Stance
Stance
Modeling Question 2)
[Walker et al. 2012, Hasan and Ng 2013]
Collective classification vs. local classification?
32
Stance
Stance
Stance
Stance
Stance
Stance
Stance
Stance
Modeling Question 2)
[Walker et al. 2012, Hasan and Ng 2013 ] [Walker et al. 2012]
Collective classification vs. local classification?
33
Stance
Stance
Stance
Stance
Disagrees
Disagrees
Disagrees
Disagrees
Jointly model disagreement together with stance?
[Abbott et. al 2012 - Linguistic Features], [Burfoot et. al 2011 for Congressional Debates]
Modeling Question 3)
34
Stance
Stance
Stance
Stance
Disagrees
Disagrees
Disagrees
Disagrees
Jointly model disagreement together with stance?
[Abbott et. al 2012 - Linguistic Features], [Burfoot et. al 2011 for Congressional Debates]
Modeling Question 3)
Stance
Stance
Stance
Stance
Disagrees
Disagrees
Disagrees
Disagrees
Stance
35
Our Contributions
• A unified framework to explore multiple models
• Fast, highly scalable inference– Large post-level graphs– Loopy author-level graphs
• Systematic study of modeling options
36
Our Contributions
• A unified framework to explore multiple models
• Fast, highly scalable inference– Large post-level graphs– Loopy author-level graphs
• Systematic study of modeling options
37
Our Contributions
• A unified framework to explore multiple models
• Fast, highly scalable inference– Large post-level graphs– Loopy author-level graphs
• Systematic study of modeling options–Modeling recommendations
38
Author
Post
Local
Collective
Joint
Author Local
Author Coll.
Author Joint
Post Local
Post Joint
Post Coll.
Modeling Granularity
Statistical Models
All Combinations of Models
39
Probabilistic Soft Logic (PSL)
• Templating language for highly scalable graphical model called Hinge-loss Markov Random Fields
Bach et. al (2015), ArXiVOpen source software: https://psl.umiacs.umd.edu
5.0: Disagrees(A1, A2) ^ Pro(A1) ~Pro(A2)
Rule Weight
40
Probabilistic Soft Logic (PSL)
• Templating language for highly scalable graphical model called Hinge-loss Markov Random Fields
5.0: Disagrees(A1, A2) ^ Pro(A1) ~Pro(A2)
Rule Weight Predicates are continuous Random Variables!
41
Probabilistic Soft Logic (PSL)
• Templating language for highly scalable graphical model called Hinge-loss Markov Random Fields
5.0: Disagrees(A1, A2) ^ Pro(A1) ~Pro(A2)
Rule Weight Predicates are continuous Random Variables!
Relaxations of Logical Operators
42
Probabilistic Soft Logic (PSL)
• Rules instantiated with variables from real network
5.0: Disagrees(A1, A2) ^ Pro(A1) ~Pro(A2)
Stance
Stance
Stance
Stance
Disagrees
Disagrees
Disagrees
Disagrees
43
Probabilistic Soft Logic (PSL)
• Rules instantiated with variables from real network
5.0: Disagrees( , ) ^ Pro( ) ~Pro( )
5.0: Disagrees( , ) ^ Pro( ) ~Pro( )
…
Continuous Random Variables!
44
Hinge-loss MRFs Over Continuous Variables
Bach et al. NIPS 12, Bach et al. UAI 13Bach et al. (2015), ArXiV
Conditional random field over continuous RVs in
[0,1]
45
Hinge-loss MRFs Over Continuous Variables
Bach et al. NIPS 12, Bach et al. UAI 13
Conditional random field over continuous RVs in
[0,1]
Feature function for each
instantiated rule
5.0: Disagrees( , ) ^ Pro( ) ~Pro( )
46
Hinge-loss MRFs Over Continuous Variables
Bach et al. NIPS 12, Bach et al. UAI 13
Conditional random field over continuous RVs in
[0,1]
Feature functions are
hinge-loss functions
47
Hinge-loss MRFs Over Continuous Variables
Bach et al. NIPS 12, Bach et al. UAI 13
Encodes distance to satisfaction of each instantiated rule
Linear function
48
Fast Inference in Hinge-loss MRFs
Bach et al. NIPS 12, Bach et al. UAI 13
Convex, continuous inference objective…
Convex optimization!
Solved using efficient, parallelizable algorithm:
Alternating Direction Method of Multipliers (ADMM)
49
Unigrams, Bigrams, Lengths, Initial n-grams, Repeated Punctuation
Obama
Bush
believe
Constructing Local Predictors
Bag-of-words
50
Unigrams, Bigrams, Lengths, Initial n-grams, Repeated Punctuation
Logistic Regression
Obama
Bush
believe
Constructing Local Predictors
Pro
Not Pro
Bag-of-words
Training Labels
51
Unigrams, Bigrams, Lengths, Initial n-grams, Repeated Punctuation
Logistic Regression
Observed Prediction Probabilities
Obama
Bush
believe
Constructing Local Predictors
Pro
Not Pro
Bag-of-words
Training Labels
LocalPro: 0.8
LocalPro: 0.1
52
PSL Rules Shared by All Models
Stance
StanceLocalPro: 0.8
LocalPro: 0.1
53
PSL Rules for Simple Collective Models
Stance
Stance
Disagrees: 1.0
LocalPro: 0.8
LocalPro: 0.1
54
Joint Disagreement Models
Stance
Stance
Disagrees
LocalPro: 0.8
LocalPro: 0.1
LocalDis: 0.5
55
Joint Disagreement Models
Stance
Stance
Disagrees
LocalPro: 0.8
LocalPro: 0.1
LocalDis: 0.5
56
Details 4Forums.com CreateDebate.org
Topics Abortion, Gay Marriage, Evolution, Gun Control
Abortion, Gay Rights, Obama, Marijuana
Avg. Users/Topic
336 311
Avg. Posts/User
19 4
Evaluation - Datasets
57
Evaluation Settings
• Prediction Tasks: Author Stance, Post Stance
58
Evaluation Settings
• Prediction Tasks: Author Stance, Post Stance
• Ground Truth for CreateDebate.org:
Stance
StanceStanceStance
Majority
59
Evaluation Settings
• Prediction Tasks: Author Stance, Post Stance
• Ground Truth for CreateDebate.org:
Stance
StanceStanceStance
Majority
• Ground Truth for 4Forums:Stanc
eStanc
e
60
Post Local
PostColl.
Post Joint
AuthorLocal
AuthorColl.
AuthorJoint
Accuracy
Author Stance Prediction – CreateDebate.org
61
Post Local
PostColl.
Post Joint
AuthorLocal
AuthorColl.
AuthorJoint
Accuracy
Author Stance Prediction – CreateDebate.org
Post < Author
62
Post Local
PostColl.
Post Joint
AuthorLocal
AuthorColl.
AuthorJoint
Accuracy
Author Stance Prediction – CreateDebate.org
Post < Author
Author-JointModel is best
63
Accuracy
Post Local
PostColl.
Post Joint
AuthorLocal
AuthorColl.
AuthorJoint
Post Stance Prediction – CreateDebate.org
Post < Author(still!)
Author-JointModel still best!
64
Post Local
PostColl.
Post Joint
AuthorLocal
AuthorColl.
AuthorJoint
Accuracy
Author Stance Prediction – CreateDebate.org
Local < Collective < Joint
65
Post Local
PostColl.
Post Joint
AuthorLocal
AuthorColl.
AuthorJoint
Accuracy
Naïve collectiveharmful at author level!
Author Stance Prediction – 4Forums.com
66
Post Local
PostColl.
Post Joint
AuthorLocal
AuthorColl.
AuthorJoint
Accuracy
Author Stance Prediction – 4Forums.com
Naïve collectiveharmful at author level!
67
Details 4Forums.com CreateDebate.org% Opposite Stance Posts
71.6 73.9
% Opposite Stance Authors
52.0 68.9
Explanation for Naïve Collective’s Performance
68
Details 4Forums.com CreateDebate.org% Opposite Stance Posts
71.6 73.9
% Opposite Stance Authors
52.0 68.9
Explanation for Naïve Collective’s Performance
Naïve collective assumptionmostly true for posts
69
Details 4Forums.com CreateDebate.org% Opposite Stance Posts
71.6 73.9
% Opposite Stance Authors
52.0 68.9
Explanation for Naïve Collective’s Performance
Naïve collective assumptionmostly true for posts
Assumption doesn’t holdat author level!
70
I agree with everything except the
last part. Safe gun storage is very
important…
I can agree with this. And in case it seemed
otherwise,I know full well how to
store guns safely…
My point was that I don’t like the idea of such a
law…
Benefit of Disagreement Prediction
Agree
Anti Gun Control
Anti Gun Control
71
I agree with everything except the
last part. Safe gun storage is very
important…
I can agree with this. And in case it seemed
otherwise,I know full well how to
store guns safely…
My point was that I don’t like the idea of such a
law…
Benefit of Disagreement Prediction
Agree
Anti Gun Control
Anti Gun Control
72
Summary
• Unified modeling framework for efficiently, systematically exploring all modeling choices
• Author-level joint disagreement and stance model best, even for post-level prediction
• Disagreement model can be vital when modeling at author level
73
Summary
• Unified modeling framework for efficiently, systematically exploring all modeling choices
• Author-level joint disagreement and stance model best, even for post-level prediction
• Disagreement model can be vital when modeling at author level
74
Summary
• Unified modeling framework for efficiently, systematically exploring all modeling choices
• Author-level joint disagreement and stance model best, even for post-level prediction
• Disagreement model can be vital when modeling at author level
75
Summary
• Unified modeling framework for efficiently, systematically exploring all modeling choices
• Author-level joint disagreement and stance model best, even for post-level prediction
• Disagreement model can be vital when modeling at author level Thank you for your attention!
top related