analysis of interaction in collaborative activities; the synergo approach
DESCRIPTION
Keynote talk at INCOS 2010 Analysis of interaction in collaborative activities: the Synergo trail It provides background information on Synergo a collaborative learning environment more at hci,ece,upatras.gr/synergoTRANSCRIPT
1/60
Analysis of interaction in collaborative activities:
the Synergo trail
Nikolaos AvourisUniversity of Patras, GR
Keynote Talk
INCoS 2010 – Thessaloniki November 24th
2/60
outline- on analysis of collaboration- the synergo testbed- synergo studies- models from synergo data
3/60
On analysis of collaborative
activities
4/60
Typical analysis objectivesfocusmethod
Participant’s perceptionsInquiry methods
Interaction processQuantitative, qualitative, sequential methods
Learning outcomesPre-post testing
Collaborative technologyUsability evaluation
5/60
Focus on the interaction process
– Dillenbourg: “the basic instrument for understanding collaborative learning is understanding the interaction that takes place during a learning process”
– Koschmann: “CSCL research is not focused on instructional efficacy, but it is studying instruction as enacted practice”
6/60
Quantitative analysis• Frequency counts of events such as:
- messages posted per student per period of time- hits on particular discussion forum pages - actions taken on objects of a shared workspace- number of files read in a shared file system etc.
• Defining metrics (indicators) that combine different kinds of frequency counts
• Suitable for all kinds of collaborative learning• They can lead to models of interaction (e.g.
Social Networks etc.)
7/60
Qualitative content analysis• “Content analysis refers to any process
that is a systematic replicable technique for compressing many words of text into fewer content categories based on explicit rules of coding” (Kripendorf, 1980)
• Suitable for every means of dialogue oriented collaborative learning (synchronous & asynchronous, collocated & distant)
8/60
Content analysis models• Henri’s scheme• Garrison’s model• Gunawardena’s Interaction
Analysis Model• Language/action OCAF
9/60
Content analysis resources
• The content analysis guidebook http://academic.csuohio.edu/kneuendorf/content/
10/60
Small group synchronous interaction: Integration of dialogue and action• Treats language acts and actions taken to objects
in an integrated way• Uniform annotation (eg. the OCAF framework)• Shifts the focus to the objects of a shared
workspace• Objects have an ‘owner’ just like language acts• Can visualize uptaking actions (Suthers 05)
11/60
Dialogue: Chat tool affordances• Visual and/or auditory cues are not available• No production blocking->overlapping exchanges• Persistence of messages – substantiation of conversation• Loose inter-turn connectedness - but possibility of
simultaneous engagement in multiple threads• Verbal deixis spans throughout the whole history of
dialogue (no restricted time window is adequate for analysis)
• Posters may reply rapidly, using short messages and split long messages to increase referent/message coherency (Garcia and Jacobs 1999)
• Participants begin new topics fairly much at will in a manner that would not happen in a formal face-to-face group discussion (O’Neil & Martin, 2003)
12/60
Action: Shared Activity spaceaffordances• Feedthrough (Dix et. al., 1993)• Various degrees of coupling (Salvador
et. al., 1996)• Workspace can be used as an external
representation of the task that allows efficient nonverbal communication
• Workspace artefacts act as conversational props (Hutchins, 1990)
13/60
Types of communication acts / gestures in shared workspace
• Deictic references• Demonstrations • Manifesting actions• Visual evidence (Gutwin, Greenberg, 2002)
14/60
Grounding through actions on a workspace representation (Suthers, 2006)Sequences of actions :(1) one participant’s action in a
medium…(2) is taken up by another participant
in a manner that indicates understanding of its meaning, and
(3) the first participant signals acceptance
15/60
Merging Action and dialogue Annotated model=collection of objects (OCAF Avouris et al. 2003)
MEF = { Entities= E (ABC) = 1/EP, FA , EI
E (VELO) = 2/ EP, FA , EI E (TRUCK) = 3/FP, FI E (STOREHOUSE) = 4/FP EC, FA, FI E (STORE) = 5/FP EC, FA, FI Ε(DELIVERY)= 11/ FP, EX, FI
Relations= R (VELO-owns-SH) = 9/FPI R (VELO-owns-ST) = 10/FPI R(TRUCK-transports- DELIVERY)=17/ EP, FI, EC R(SH-are-suppplied-by-TR) = 18/ FIM R (ABC-owns-TR) = 25/ FPI R(ST-owns-SH) = 24/ EP FP FI EC, EM R (ABC-owns-TR) = 25/ FPI
Attributes= A (DEL.id) = 13/FIM A (DEL.volume) = 14/FIM A (DEL.Weight) = 15/FI A (DEL.Destination) = 16/FI A (TR.Max_Weight ) = 19/FI A (TR.id ) = 21/EP , FI A (TR.Journey_id ) = 23/FI A (TR.volume ) = 20FIM A (SH.id ) = 24/FI
Items not in the final solution -R (SH-DEL) = 12/EP , FR , -A(VELO.Storehouse)=6/ EP , FC -A(VELO.Store)= 7/ EP , FC -A(ABC.Truck)= 8/ FP , EX -A (TR.max_journeys_per_week) = 22/EP , FR }
A(volume)
A(destination)
A(Journey _id)
A(id) A(volume)
E(VELO)
2/EP, FA , EI
E(ABC) 1/EP, FA , EI
E(STORE-HOUSE)
4/FP EC, FI
E(STORE)
5/FP , EC, FAI
E(TRUCK) EP, FI
E(DE-LIVERY)
11/FP, EX, FI
20/FI,M
23/FI
21/EP, FI
14/FIM
16/FI
R
9/FPI
R 24/EP FPI, EM
A(id)
24/ FI
R
10/FPI
R 18/FIM
R 25/FPI
R 17/EP,FI,EC
R 12/EP, FR
A(Max_ weight)
19/FI
A(Id)
13/FIM
A(Weight
15/FI
A (max-journeys/week 22/EP, FR
A (storehouse)
6/EP, FC
A (store)
7/EP, FC A (truck) 8/FP, EX
16/60
Synergo
Avouris et al. 2004hci.ece.upatras.gr/synergo
Chat
Act
17/60
Synergo
Chat tool
Shared Activity Space
Drawing objects libraries
Partner selectiontool
18/60
Synergo Drawing librariesConcept maps
Flow charts
Entity-Relationship Diagrams
Free Drawing
19/60
Activity logging
used for :• Build a history of interaction at server• support latecomers during synchronous collaboration• analysis and playback of the activity •Support replication/ reduce bandwidth requirements
20/60
Analysis tools
20
21/60
Log Data Preprocessor
21
Analysistools
22/60
Typed events automatically annotate the diagram
Object A
I C
Actor A Actor B Actor C
Types of events I (Insert), M (Modify), D (Delete) C (Contest)
M R
( )itoaii TOAtE ][],[,,=
23/60
Playback of annotated view
24/60
What about the chat? Can we annotate chat automatically?
One approach is to ask the user to do it - open sentences (e.g. Epsilon (Soller et al. 97)
25/60
Abstract objects
Dialogue messages
Model objects
Deleted objects
(b)
Annotation of chat events
26/60
Define types of actions (annotation scheme)
27/60
Overview: Visualization of logged actions
28/60
Teachers view and tool support
• E. Voyiatzaki, M. Margaritis, N. Avouris, Collaborative Interaction Analysis: The teachers' perspective, Proc.ICALT 2006 - The 6th IEEE International Conference on Advanced LearningTechnologies. July 5-7, 2006 – Kerkrade , Netherlands, pp. 345-349.
29/60
Teacher support (supervisor tools)
30/60
Study of the use of tools by teachersComputer Engineering University degree program (A’ Semester)
Level of Education
1 Teacher + 5 Teaching Assistants Teachers involved
80 students (46 students 2004-2005, 34 students 2005-2006)
Learners involved
Problem solving activity: Development and Exploration of an Algorithm
Students in Dyads , no roles assignedTypical Laboratory lesson (2 didactic hours)
Collaborative Activity
SYNERGO Collaborative EnvironmentSYNERGO Analysis Tools
Collaborative Tools
31/60
teacher
The Teachers Used the proposed views and gave feedback…
Quantified Overview:Class and
Group
The ProcessView
(Playbackof the
activity)
Qualitativeview
Rowdata
researcher
Teachers: “The process view is the most important tool for in depth insight .”
32/60
studies
Vrachneika Gymnasio-3rd year
UnivPatras Algorithms
33/60
Typical tasks- Collaborative Cognitive Walkthrough of an interactive system
- Designing Data bases (ER-D)
- Building and exploring Flow Charts
34/60
Joint Univ Patras -UnivDuisburg croos-national collaborative activities (2004-2005)
• A. Harrer, G. Kahrimanis, S. Zeini, L. Bollen, N. Avouris, Is there a way to e-Bologna? Cross-National Collaborative Activities inUniversity Courses, Proceedings EC-TEL, Crete, October 1-4, 2006, LNCS vol. 4227/2006, pp. 140-154, Springer Berlin
35/60
Similar models with different tools (Synergo, Freestyler)
36/60
Findings of the Patras-Duisburg study
•Mixture of synchronous and asynchronous approaches.
•Only partly use of the provided tools •Engaging activities - examples of sessions of many hours (4-5 h) in joint activity and discussion
• Innovative use of media and coordination mechanisms
•Good strategies for division of labor•Excellent social dynamics and group spirit.
37/60
A distance learning course of Hellenic Open University (HOU) (2003-2004)
M. Xenos, N. Avouris, D. Stavrinoudis, and M. Margaritis, Introduction of synchronous peer collaboration activities ina distance learning course, IEEE Transactions inEducation, vol. 52 ( 3), Aug. 2009, pp. 305 - 311,
38/60
Synergo server
ODL Server (forum, exchange of material,
help desk) Asynchronous interaction
Synchronous interaction (share
drawing / chat communication)
Synergo client
Synergo client
ODL repository
Activity logging
Submit final solution Record
activity
Student #1 Student #2
Post assignments, form groups
Tutor
Facilitator
Arrangements on sessions plan- direct contact
Respond to technical and organizational problems –
follow activity
GroupGroup
Mixed media and collaboration approachesAsynchronous group activity
Synchronous activity
39/60
Synergo- Discussion forum
40/60
Findings of the HOU study
• Infrastructure overhead higher than expected (unforeseen technical problems)
• Peer tutoring patterns emerged in higher degree than younger students
• Multiple media engaged• Strong social aspects of community
building
41/60
Study on Mecahnics of Collaboration:Coordination protocol
Group B No floor control: all partners can act in the shared work space
Group A Explicit floor control: Only the key owner can act in the shared work space
0
20
40
60
80
100
120
140
160
180
200
Critical Insert Delete Move Chats
Type of events
Num
ber o
f eve
nts
GROUP A (with key)
GROUP Β (without key)
T+ T-
42/60
Findings of the study§ Explicit floor control of the shared activity area did not inhibit problem solving
§ Similar patterns of activity in both groups.
§ group T- was more active than T+
§ T+ students have been obliged to negotiate on possession of the activity enabling key and thus argue at the meta-cognitive level of the activity and externalise their strategies, a fact that helpedthem deepen their collaboration
43/60
models
44/60
#1 Support for Group Awareness through a Machine Learning ApproachTrain a classifier to be used for estimation of the quality of collaboration using historical data of problem solving activities of students engaged in building concept maps and flow-chart diagrams in Hellenic Open University and University of PatrasM. Margaritis, N. Avouris, G. Kahrimanis, On Supporting Users’Reflection during Small Groups Synchronous Collaboration, 12th International Workshop on Groupware, CRIWG 2006 Valladolid, Spain, September 17-21, 2006, LNCS 4154
45/60
Logfile segmentation L={S1, S2, … Sk}
NE
quality of collaboration per segment (bad, average, good)
46/60
Correlation based feature selection(CFS) for different segment sizes
NE=60 NE=80 NE=100 NE=200 (2) num_chat (2) num_chat (2) num_chat (2) num_chat
(3)symmetry_chat (3)symmetry_chat
(4) altern_chat (4) altern_chat (4) altern_chat (4) altern_chat
(5) avg_words (5) avg_words (5) avg_words (5) avg_words
(6) num_quest (6) num_quest (6) num_quest
(7) num_draw (7) num_draw (7) num_draw (7) num_draw Correlation based Feature Selection (CFS)
technique:
makes use of a heuristic algorithm alongwith a gain function to validate theeffectiveness of feature subsets.
NE= number of events per segment
47/60
Performance of classificationalgorithms
• Naïve BayesianNetwork
• Logistic Regression• Bagging• Decision Trees• Nearest Neighbor 75
80
85
90
60 80 100 200Fragmentation factor NE
Suc
cess
rate
(%)
NaiveBayesLogisticBaggingSimpleLogisticRandomForestNNge
48/60
Visualization of group awareness indicator
State of Collaboration
49/60
Evaluation study• 11 groups of 3 students each were given a
collaborative task. • 6 of these groups were provided with the group
awareness mechanibsm. • 5 groups did not have that facility• The mean values of collaboration symmetry
were significanlty different between the two sets (p=0,0423).
50/60
Side-effect
• in four (4) out of the six (6) groups therewas an explicit discussion about the groupawareness mechanism.
• A side-effect:
51/60
#2 Measuring quality of collaboration in Synergoactivities using a rating scheme and an automatic rating model
Based on: Meier, A., Spada, H., & Rummel, N. (2007). A rating scheme for assessing the quality of computer-supported collaboration processes. International Journal of Computer-Supported Collaborative Learning, 2, 63–86.
52/60
Original setting New setting
Desktop-videoconferencingsystem with shared texteditor
Synergo: shared whiteboardand chat
Medical decision making Computer programming(algorithm building)
Intermediates;complementary prior
knowledge (psychology andmedicine)
Beginners;similar prior knowledge
CSCL tool
Domain
Collaborators
Meier et al. (2007) rating scheme
53/60
Meier et al (2007) rating scheme dimensions
54/60
Kahrimanis et al. (2009) adapted collaboration rating scheme
7. Individual Task Orientation (for dyad mean or absolute difference)
Motivation
6 .Cooperative Orientation Interpersonal Relationship
5 .Structuring the Problem Solving ProcessCoordination4. Argumentation
3. Knowledge Exchange Joint information processing
2. Sustaining Mutual Understanding 1. Collaboration Flow Communication
Dimensions Aspect of collaboration
55/60
Development of a Collaboration Quality Estimation Model
Data set used• 350 students of 1st year working in
dyads to solve an algorithmic problem using Synergo (academic year 2007-2008) duration of activity 45’ to 75’
• 260 collaborative sessions• Grading according to the quality of
solution and quality of collaboration
56/60
36 derived metrics used(Kahrimanis et al. 2010)
57/60
Quality of Collaboration Estimator(Kahrimanis et al. 2010)
Observed vs. Estimated CQ average
-2
-1
0
1
2
3
-2 -1 0 1 2 3
Estimated(collaboration quality avg)
Obs
erve
d (c
olla
bora
tion
qual
ity a
vg)
VIPs (1 Comp / 95% conf. interval)
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
Variable
VIP
collaboration_quality_avg = 0.460 + 0.004*C4 - 0.005*C6 + 0.011*C8_17.5 - 0.012*C7
+ 0.602*EV3 + 0.447*STC - 0.001*MW1 + 0.008*MW6
Stone & GeisserCoefficient
(cross validation)
Partial Least Squares Regression Model
58/60
Use of Quality of Collaboration Estimator as discriminator between cases of good and bad collaboration
• The model scored between 76.6% to 79.2%, with the exception of one dimension of lower quality.
59/60
Use of Quality of Collaboration Estimator as automatic rater
• The model had acceptable performance as rater as the inter-rater reliability with human raters had the following values: ICC=.54, Cronbach’s α=.70, Spearman’s ρ=.62 acceptable for α και ρ (George, & Mallery, 2003; Garson, 2009), not for ICC(.7) (Wirtz & Caspar, 2002) . This applies both for the average collaboration quality value and the individual dimensions.
60/60
Current developments• Study of tablet-based collaboration patterns
(synergo v. 5)
• Study of Attention mechanisms (Chounta et al. 2010)
61/60
More on Synergo:hci.ece.upatras.gr/synergo
62/60
Some more key references• Avouris N., Margaritis M., & Komis V. (2004). Modelling interaction
during small-group synchronous problem-solving activities: TheSynergo approach, 2nd Int. Workshop on Designing ComputationalModels of Collaborative Learning Interaction, ITS2004, Maceio, Brasil, September 2004.
• Κahrimanis, G., Meier, A., Chounta, I.A., Voyiatzaki, E., Spada, H., Rummel, N., & Avouris, N. (2009). Assessing collaboration quality insynchronous CSCL problem-solving activities: Adaptation andempirical evaluation of a rating scheme. Lecture Notes in ComputerScience, 5794/2009, 267-272, Berlin: Springer-Verlag.
• Kahrimanis G., Chounta I.A., Avouris N., (2010) Determiningrelations between core dimensions of collaboration quality - A multidimensional scaling approach, In the 2nd InternationalConference on Intelligent Networking and Collaborative Systems(INCoS 2010)