odr 2013 sdskills dashboard umass

An Online Deliberation Facilitators' Dashboard:

Visualizations and Text analysis to support quality dialogues

Wing, L., Murray, T., Woolf, B., Katsh, E. The Twelfth International Online Dispute Resolution Forum,

Montreal, June 2013

2

“The Fourth Party: Improving Computer-Mediated Deliberation through Cognitive, Social and Emotional

Support”

• 3-Year NSF Social Computing grant, started Fall 2010

• Description at www.socialdeliberativeskills.com

3

Project collaborators

• Beverly Woolf: CompSci, PI (intelligent and collaborative educational systems)

• Tom Murray: CompSci; project manager/co-PI, principal visionary and instigator (ed-tech, cog-psych & D&D)

• Ethan Katsh (ODR), Legal Studies, co-PI• Leah Wing (social justice and ODR), Legal Studies, co-PI• Linda Tropp, Psychology of Peace and Violence, advisor (intergroup

relations/conflict) • Zan Goncalves, New England Center for Civic Life (teaching,

practice and study of deliberative democracy)• Idealogue Inc.; iCohere. Inc. software platforms(Advanced

dialogue)

4

Thanks for consultation and/or data from:

• DemarsAssociates.com/PayPal/ebay (e-commerce)• Juripax.com (online workplace and divorce

settlements)• National Mediation Board (transportation

management/labor disputes)• Modria.com (e-commerce plus)• Idealogue.com (depth-oriented online dialogue

platform)• iCohere.com (online communities and work groups)• Mass Dept. of Dispute Resolution (civic engagement)• New England Center for Civic Life (teaching, practice

and study of deliberative democracy)

5

Social Deliberative Skills:Social/Emotional/Reflective

Perspective taking & cognitive empathy

Perspective seeking (curiosity/inquiry)

Self-reflection: on one's biases, intentions, emotional state

Meta-dialog: Reflect on the quality of the dialog

Epistemic skill: e.g. treating facts/data differently from opinions/hypotheses

Tolerance for uncertainty, ambiguity, disagreement, paradox

…

Support/Scaffolding vs. “Education”

FacilitatedOnline

DELIBERATION

Outcomes:- Agreements/solutions

- Relationship, Trust (social capital), understanding

- SKILL USE (and practice)

ExistingSkills

Adaptive Support(4th party)

Interface Support

FacilitatorSupport

(Dashboard)

7

Example: Topics Chosen by Students—set context in Spring ’12

• Week 1: Discuss the pros and cons of legalizing marijuana

• Week 2: Sex – what's the big deal? What values are most important in making sexual choices?

• Week 3: Discus the pros and cons of the death penalty (capital punishment)

8

9

Mediem

Opinion Sliders

10

Experimental ConditionsExp Group N Gender Grade

Vanilla 8 (5 Female, 3 Male)

4 soph, 4 juniors, 0 seniors

Reflective Tools 8 (5 Female, 3 Male)

4 soph, 2 juniors, 2 seniors

(Sliders) 8 (Group omitted due to interaction issues)

• V&R groups: 241 posts and 516 segments (average of 15.06 (SD = 7.45) posts/student)

• Mean words/post = 54 (SD = 42); mean characters/post = 299 (SD = 242)

11

Text Coding

12

Main Effect Exp. Group Total_

SD_SkillIntersubjectivespeech acts

Vanilla (N = 8) 0.29 (0.07) 0.20 (0.09)

Reflective Tools (N = 8) 0.40 (0.08) 0.30 (0.08)

• A significant difference and main effect between Total-SD-Score and grouping, F(1, 14) = 6.89, p = 0.02*, d = 1.46 (a large effect) in favor of the Reflective Tools group

• A significant relationship between Intersub and grouping, F(1, 14) = 4.81, p = 0.05*, d = 1.05 (a large effect) in favor of the Reflective Tools group

13

Facilitator’s Dashboard

14

Dashboard Text Tagging

Advice Screen

Applicability and Use

Facilitators• To identify individual and group participation levels

for assessing useful interventions(ie: to stimulate more involvement, moreconstructive engagement, etc.)

• To identify patterns of ‘silence’ or intensity (ie: to ID topics/relationships to attend to; use as clues to search text analysis forconflicts or breakthroughs)

Applicability and Use

Participants• For self/group assessment of engagement,

topical, and relationship patterns re:

participation, intensity, silence• For clues to search text analysis for

conflicts or breakthroughs

18

CohMetrixdiscourse & coherence

LIWClexical categories

Skill vs. text metrics correlations

LIWC/Cohmetrix Correlations with Total-SD-Skill

Cross-Domain Training

Prelim. Machine Learning ResultsFor Predicting Total Skill

Future ApplicationsCommon problems encountered in online facilitation

• Low or no participation of individuals or groups, or silences or lulls on the part of individuals, the entire group, or sub-groups

• Conversation domination by an individual or group • Inappropriate or disrespectful behavior • Off-topic conversation • Tension-filled disagreements, or high emotional content• Too much agreement or politeness • Misunderstanding due to missing communication skills

normally available in face-to-face communication• Violation of rules (e.g. confidentiality, no advertising, etc.)

Thank you

Extra slides

Automated Text Analysis

LIWC (Pennebaker et al.) – Dictionary-based (linguistic inquiry word count)

– 4,500 words/STEMS; 80 word categories– we focus on 19 of them– 80 >> 4 general descriptor categories (word count, words per sentence, % of words captured, and % of words >6 letters),

22 standard linguistic dimensions (e.g., % pronouns, articles, auxiliary verbs, etc.), 32 psychological constructs (e.g., affect, cognition, biological processes), 7 personal concern categories (e.g., work, home, leisure activities), 3 paralinguistic dimensions (assents, fillers, nonfluencies), and 12 punctuation categories (periods, commas, etc).

– Relate the categories to things like aggression, used in theraputic contexts, to ID lying,

• Coh-Metrix (Graesser et al.)– syntax, referential cohesion, semantic cohesion, rhetorical composition…

– 100 measurements categories– We focus on 4 composite measurements (or major factors):

Narrativity, Referential Cohesion, Syntactic Simplicity, and Word Concreteness

27

Experimental Groups

28

Code Frequencies  Intersub Meta_

DialogueMeta_Topic

Apology Appreciation

Fact_Source

Source_Ref

#students

22 5 15 1 8 1 4

% ofsegs 25% 0.9% 5.5% 0.2% 1.3% 0.3% 1.2%

29

Total Skill score adds:

• Appreciation (Gratitude, affirmation of another's idea or situation)

• Apology• Fact--sourced (stating a fact and noting the source in the

same post)• Source Reference (Mentioning a source, with a

reference or description; without a fact)• Intersubjectivity: perspective taking or question asking• Meta-dialogue, discussing the quality of the dialogue• Meta-Topic: Birds eye or systemic view of the topic

Next: Linked Representations• clicking on the name of an individual or group in a chart or network

diagram will focus (or filer or highlight) all tools on that individual or group;

• clicking on a link in the network diagram will show posts between the relevant interlocutors;

• clicking on a word in the word cloud will highlight posts including that word;

• clicking on a location in the time-axis of a trend line will navigate to posts in the Timeline at that time;

• hovering over an agent trigger will show the Advice or Alert associated with that event; and clicking on Advice or Alerts will navigate to the place in the dialogue timeline for the triggering event(s).

31

Text Coding

32

Social Deliberative Skills:Social/Emotional/Reflective

• 1. Social perspective taking (cognitive empathy, reciprocal role taking...)

• 2. Social perspective seeking (social inquiry, question asking skills...)

• 3. Social perspective monitoring (self-reflection, meta-dialogue...)

• 4. Social perspective weighing (reflective reasoning; comparing and contrasting views...)

Domain statistics and inter-rater agreement

34

Codoole—coding toolsCoole – coding tools

Social Deliberative Skill:application of HOSs to me/you/we

Higher Order Skills • argumentation• critical thinking• explanation & clarification• inquiry/curiosity (question asking & investigation)• reflective judgment• meta-cognition• epistemic reasoning

Apply these skills, not to EXTERNAL REALITY (“IT”/problem domain) but to theINTERSUBJECTIVE domain

Higher Order Skills applied to:

SELFgoals; level of certainty; feelings, values, assumptions…

YOU goals, assumptions, feelings, values; perspective taking; "believing" & cognitive empathy…

WEagreements, goals; quality of the discourse/collaboration; differences and similarities in values, beliefs, goals, power, roles…

Survey Questions

29 of the 36 participants took the survey

By Demographic

Settings

Post number vs. size