odr 2013 sdskills dashboard umass
DESCRIPTION
TRANSCRIPT
An Online Deliberation Facilitators' Dashboard:
Visualizations and Text analysis to support quality dialogues
Wing, L., Murray, T., Woolf, B., Katsh, E. The Twelfth International Online Dispute Resolution Forum,
Montreal, June 2013
2
“The Fourth Party: Improving Computer-Mediated Deliberation through Cognitive, Social and Emotional
Support”
• 3-Year NSF Social Computing grant, started Fall 2010
• Description at www.socialdeliberativeskills.com
3
Project collaborators
• Beverly Woolf: CompSci, PI (intelligent and collaborative educational systems)
• Tom Murray: CompSci; project manager/co-PI, principal visionary and instigator (ed-tech, cog-psych & D&D)
• Ethan Katsh (ODR), Legal Studies, co-PI• Leah Wing (social justice and ODR), Legal Studies, co-PI• Linda Tropp, Psychology of Peace and Violence, advisor (intergroup
relations/conflict) • Zan Goncalves, New England Center for Civic Life (teaching,
practice and study of deliberative democracy)• Idealogue Inc.; iCohere. Inc. software platforms(Advanced
dialogue)
4
Thanks for consultation and/or data from:
• DemarsAssociates.com/PayPal/ebay (e-commerce)• Juripax.com (online workplace and divorce
settlements)• National Mediation Board (transportation
management/labor disputes)• Modria.com (e-commerce plus)• Idealogue.com (depth-oriented online dialogue
platform)• iCohere.com (online communities and work groups)• Mass Dept. of Dispute Resolution (civic engagement)• New England Center for Civic Life (teaching, practice
and study of deliberative democracy)
5
Social Deliberative Skills:Social/Emotional/Reflective
Perspective taking & cognitive empathy
Perspective seeking (curiosity/inquiry)
Self-reflection: on one's biases, intentions, emotional state
Meta-dialog: Reflect on the quality of the dialog
Epistemic skill: e.g. treating facts/data differently from opinions/hypotheses
Tolerance for uncertainty, ambiguity, disagreement, paradox
…
Support/Scaffolding vs. “Education”
FacilitatedOnline
DELIBERATION
Outcomes:- Agreements/solutions
- Relationship, Trust (social capital), understanding
- SKILL USE (and practice)
ExistingSkills
Adaptive Support(4th party)
Interface Support
FacilitatorSupport
(Dashboard)
7
Example: Topics Chosen by Students—set context in Spring ’12
• Week 1: Discuss the pros and cons of legalizing marijuana
• Week 2: Sex – what's the big deal? What values are most important in making sexual choices?
• Week 3: Discus the pros and cons of the death penalty (capital punishment)
8
9
Mediem
Opinion Sliders
10
Experimental ConditionsExp Group N Gender Grade
Vanilla 8 (5 Female, 3 Male)
4 soph, 4 juniors, 0 seniors
Reflective Tools 8 (5 Female, 3 Male)
4 soph, 2 juniors, 2 seniors
(Sliders) 8 (Group omitted due to interaction issues)
• V&R groups: 241 posts and 516 segments (average of 15.06 (SD = 7.45) posts/student)
• Mean words/post = 54 (SD = 42); mean characters/post = 299 (SD = 242)
11
Text Coding
12
Main Effect Exp. Group Total_
SD_SkillIntersubjectivespeech acts
Vanilla (N = 8) 0.29 (0.07) 0.20 (0.09)
Reflective Tools (N = 8) 0.40 (0.08) 0.30 (0.08)
• A significant difference and main effect between Total-SD-Score and grouping, F(1, 14) = 6.89, p = 0.02*, d = 1.46 (a large effect) in favor of the Reflective Tools group
• A significant relationship between Intersub and grouping, F(1, 14) = 4.81, p = 0.05*, d = 1.05 (a large effect) in favor of the Reflective Tools group
13
Facilitator’s Dashboard
14
Dashboard Text Tagging
Advice Screen
Applicability and Use
Facilitators• To identify individual and group participation levels
for assessing useful interventions(ie: to stimulate more involvement, moreconstructive engagement, etc.)
• To identify patterns of ‘silence’ or intensity (ie: to ID topics/relationships to attend to; use as clues to search text analysis forconflicts or breakthroughs)
Applicability and Use
Participants• For self/group assessment of engagement,
topical, and relationship patterns re:
participation, intensity, silence• For clues to search text analysis for
conflicts or breakthroughs
18
CohMetrixdiscourse & coherence
LIWClexical categories
Skill vs. text metrics correlations
LIWC/Cohmetrix Correlations with Total-SD-Skill
Cross-Domain Training
Prelim. Machine Learning ResultsFor Predicting Total Skill
Future ApplicationsCommon problems encountered in online facilitation
• Low or no participation of individuals or groups, or silences or lulls on the part of individuals, the entire group, or sub-groups
• Conversation domination by an individual or group • Inappropriate or disrespectful behavior • Off-topic conversation • Tension-filled disagreements, or high emotional content• Too much agreement or politeness • Misunderstanding due to missing communication skills
normally available in face-to-face communication• Violation of rules (e.g. confidentiality, no advertising, etc.)
Thank you
Extra slides
Automated Text Analysis
LIWC (Pennebaker et al.) – Dictionary-based (linguistic inquiry word count)
– 4,500 words/STEMS; 80 word categories– we focus on 19 of them– 80 >> 4 general descriptor categories (word count, words per sentence, % of words captured, and % of words >6 letters),
22 standard linguistic dimensions (e.g., % pronouns, articles, auxiliary verbs, etc.), 32 psychological constructs (e.g., affect, cognition, biological processes), 7 personal concern categories (e.g., work, home, leisure activities), 3 paralinguistic dimensions (assents, fillers, nonfluencies), and 12 punctuation categories (periods, commas, etc).
– Relate the categories to things like aggression, used in theraputic contexts, to ID lying,
• Coh-Metrix (Graesser et al.)– syntax, referential cohesion, semantic cohesion, rhetorical composition…
– 100 measurements categories– We focus on 4 composite measurements (or major factors):
Narrativity, Referential Cohesion, Syntactic Simplicity, and Word Concreteness
27
Experimental Groups
28
Code Frequencies Intersub Meta_
DialogueMeta_Topic
Apology Appreciation
Fact_Source
Source_Ref
#students
22 5 15 1 8 1 4
% ofsegs 25% 0.9% 5.5% 0.2% 1.3% 0.3% 1.2%
29
Total Skill score adds:
• Appreciation (Gratitude, affirmation of another's idea or situation)
• Apology• Fact--sourced (stating a fact and noting the source in the
same post)• Source Reference (Mentioning a source, with a
reference or description; without a fact)• Intersubjectivity: perspective taking or question asking• Meta-dialogue, discussing the quality of the dialogue• Meta-Topic: Birds eye or systemic view of the topic
Next: Linked Representations• clicking on the name of an individual or group in a chart or network
diagram will focus (or filer or highlight) all tools on that individual or group;
• clicking on a link in the network diagram will show posts between the relevant interlocutors;
• clicking on a word in the word cloud will highlight posts including that word;
• clicking on a location in the time-axis of a trend line will navigate to posts in the Timeline at that time;
• hovering over an agent trigger will show the Advice or Alert associated with that event; and clicking on Advice or Alerts will navigate to the place in the dialogue timeline for the triggering event(s).
31
Text Coding
32
Social Deliberative Skills:Social/Emotional/Reflective
• 1. Social perspective taking (cognitive empathy, reciprocal role taking...)
• 2. Social perspective seeking (social inquiry, question asking skills...)
• 3. Social perspective monitoring (self-reflection, meta-dialogue...)
• 4. Social perspective weighing (reflective reasoning; comparing and contrasting views...)
Domain statistics and inter-rater agreement
34
Codoole—coding toolsCoole – coding tools
Social Deliberative Skill:application of HOSs to me/you/we
Higher Order Skills • argumentation• critical thinking• explanation & clarification• inquiry/curiosity (question asking & investigation)• reflective judgment• meta-cognition• epistemic reasoning
Apply these skills, not to EXTERNAL REALITY (“IT”/problem domain) but to theINTERSUBJECTIVE domain
Higher Order Skills applied to:
SELFgoals; level of certainty; feelings, values, assumptions…
YOU goals, assumptions, feelings, values; perspective taking; "believing" & cognitive empathy…
WEagreements, goals; quality of the discourse/collaboration; differences and similarities in values, beliefs, goals, power, roles…
Survey Questions
29 of the 36 participants took the survey
By Demographic
Settings
Post number vs. size