an experience of peer evaluation in a b-learning environment
DESCRIPTION
An experience of peer evaluation in a b-learning environment. EDMEDIA 2014, 23 to 26 juin 2014 Tampere-Finland. The Kelluwen b-learning community. Who participate at Kelluwen ? Teachers and students open to innovation processes in their classes From 2010 we work with a community - PowerPoint PPT PresentationTRANSCRIPT
An experience of peer evaluation
in a b-learning environmentEliana Scheihing
Univ. Austral of [email protected]
Julio Daniel GuerraUniv. Austral of Chile
& University of [email protected]
Sergio BustamanteUniv Austral of Chile
EDMEDIA 2014, 23 to 26 juin 2014 Tampere-Finland
The Kelluwen b-learning community
Who participate at Kelluwen?
Teachers and students open to innovation processes in their classes
From 2010 we work with a community 3 regions (Los Ríos, Los Lagos y Aysén) 17 cities 57 schools 93 teachers and 4517 students 160 classes from Valdivia to Aysén
Which is the proposal of Kelluwen?
Social Web
Socio-communicative
skills
Collaborative learning
Which is the opportunity that address Kelluwen?
Motivation of the students with the Social Web tools
Large inversion in technologic infrastructure in the schools but underused
Low development of socio-communicative skills in the students
How we contribute to the scholar context?
Motivating the creation of a learning
community
Coordinating scholar networks by means of a didactic proposal (Didactic design) and a supporting Web platform
The peer review module in Kelluwen platform
The Didactic Designs (DDs) consider students working in teams within each classroom
For some activities within the DDs, we promote that teams from different schools and different geographical locations do peer review activities
Coordination between different schools is challenging, and to support this process we implemented a peer review module, called Works Tool
The peer review module in Kelluwen platform: teacher’s view
The peer review module in Kelluwen platform: teacher’s view of reviews
Imagen con la pauta de evaluación
An example of peer review
Qualitative study of the peer review module
Sample of the survey questions for students (top) and teachers (bottom).
Classroom and data
Class School Nº of Students
Nº of teams
Nº of Publications
Nº of received reviews
Nº of sent reviews
1ºA Colegio Adventista
39 10 32 17 20
1ºA Liceo Aysén 42 18 54 15 0
1ºC Liceo Inglés 31 6 13 7 12
1ºB Colegio Adventista
45 9 23 22 28
1ºA Colegio Helvecia
37 12 21 23 24
Totals 194 55 143 84 84
Didactic Design 21: Literary and non-literary in Youtube
Classroom and data
Class School Nº of Students
Nº of teams
Nº of Publications
Nº of received reviews
Nº of sended reviews
1ºA Colegio Darío Salas 38 7 16 14 14
4ºM Liceo Río Bueno 25 5 6 10 10
1ºB Colegio San Francisco
25 5 36 10 10
1ºA Colegio Emprender 42 9 19 18 18
4ºA Liceo Rayen Mapu 36 7 14 14 14
1ºM Alianza Hospitalaria 4 1 1 2 2
4ºM Colegio Austral 20 8 13 16 16
Totals 190 42 105 84 84
Didactic Design 88: Building a slideshow of the Twentieth Century
Quantitative study of the peer review module
DD Classes Different schools
Within class reviews
Between class reviews
Valid comments
within class
Valid comments between classes
21 5 4 32 52 30 52
88 7 7 28 56 28 55
Analysis and results
DD Number of questions
Correlation of scores by question
Number of products Correlation of scores by product
21 138 0.0503 31 -0.1115
88 245 0.1161 45 0.1817
Correlations between two reviewers
Analysis and results
DD Same class Twin class p-value Wilcoxon
testNºrev Min 1st qu. Med. 3th qu. Max Nº rev Min 1st qu Med 3th qu. Max
21 32 1.25 2.47 2.9 3.75 4 52 1.2 2.4 2.75 3.05 4 0.0943
88 30 2.6 3.4 3.6 3.9 4 56 2 3 3.4 3.8 4 0.0818
p-values of the Wilcoxon test to contrast localization parameters of score distributions
H0: Ls = Lt vs H1: Ls > Lt
DD Assessment Feedback Form
21 0.879341 0.759162 0.594582
88 0.896736 0.739674 0.488020
Analyzing quality of commentsTwo independent judges evaluated the comments with respect
to three criteria: assessment, feedback, and form with a four level rubric:
Weighted Kappa statistics to measure agreement between judges
Not made Fairly achieved achieved Fully achieved
1 2 3 4
Assessment criteria Feedback criteria Form criteria
DD Judge 1 Judge 2 Judge 1 Judge 2 Judge 1 Judge 2
21 0.5377376 0.5299191 -0.1761536 -0.0591588 0.3469381 0.2030653
88 0.6024059 0.6734458 -0.4291461 -0.3331847 -0.0070753 -0.0254401
Analyzing quality of comments
Correlations between reviewers’ scores and judges’ labels
DD Criteria Same class Twin class p-valueWilcoxon test
Nº rev. Median Nº rev. Median
21 Assessment 30
3 52
2 0.0479
Feedback 3 3 0.5333
Form 3 3 0.6916
88 Assessment 28
4 55
3 0.0818
Feedback 3 3 0.7791
Form 3 3 0.8482
p-values of the Wilcoxon test to contrast localization parameters of label distributions
Analyzing comments: same class vs twin classes
Conclusions We developed a peer review module in our Web platform that support the pairing processset a non-anonymity environment where • students can review, send free text comments and discuss
among them• teacher can monitor the work
We analyze potential differences between peer reviews when they are done by students: on the same class group or from class groups of different schools.
We analyze several class groups doing review activities in which pairing resulted to be mixed.
Conclusions We found significant differences between reviewers from the same class versus those from different classes, in the review scores and in the quality of feedback
When reviewers are from the same class, they tend to give higher scores to their peers and tend to write more accurate assessment comments.
This seems to confirm the idea that peer reviews gain quality when reviewers know the reviewed students, in the sense that non anonymity make reviewers more aware of what they are reviewing.