phil davies school of computing university of glamorgan “super u”
DESCRIPTION
Peer-Assessment: No marks required, just feedback? Evaluating the Quality of Computerized Peer-Feedback compared with Computerized Peer-Marking. Phil Davies School of Computing University of Glamorgan “Super U”. Definition of Peer-Assessment?. In describing the teacher .. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/1.jpg)
11
Phil DaviesPhil DaviesSchool of ComputingSchool of Computing
University of Glamorgan “Super U”University of Glamorgan “Super U”
Peer-Assessment:Peer-Assessment: No marks required, just No marks required, just
feedback?feedback?Evaluating the Quality of Evaluating the Quality of
Computerized Peer-Feedback Computerized Peer-Feedback compared with Computerized compared with Computerized
Peer-Marking Peer-Marking
![Page 2: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/2.jpg)
22
Definition of Peer-Assessment?Definition of Peer-Assessment?
In describing the teacher ..In describing the teacher ..
A tall b******, so he was. A tall thin, mean b******, A tall b******, so he was. A tall thin, mean b******, with a baldy head like a lightbulb. He’d make us with a baldy head like a lightbulb. He’d make us mark each other’s work, then for every wrong mark each other’s work, then for every wrong mark we got, we’d get a thump. That way – he mark we got, we’d get a thump. That way – he paused – ‘we were implicated in each other’s pain’paused – ‘we were implicated in each other’s pain’
McCarthy’s Bar McCarthy’s Bar (Pete McCarthy, 2000,page 68)(Pete McCarthy, 2000,page 68)
![Page 3: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/3.jpg)
33
Why Peer-AssessmentWhy Peer-Assessment Community of Practice!!Community of Practice!! Working together to learn (and be assessed)Working together to learn (and be assessed) PerceptionsPerceptions
– Staff > Lecturer getting easy life not markingStaff > Lecturer getting easy life not marking– Student> Lecturer getting easy life not markingStudent> Lecturer getting easy life not marking
Student Awareness of Benefits …. IMPORTANT Student Awareness of Benefits …. IMPORTANT TANGIBILITYTANGIBILITY
Success dependant upon scalability Success dependant upon scalability (computerisation) .. “Student Numbers have risen (computerisation) .. “Student Numbers have risen dramatically since 1991 without a concomitant dramatically since 1991 without a concomitant increase in resources” (Pond et al, 1995)increase in resources” (Pond et al, 1995)
![Page 4: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/4.jpg)
44
Computerised Peer-AssessmentComputerised Peer-Assessment
CAP SystemCAP System
Permits students to mark & comment the work of Permits students to mark & comment the work of other students . (normally 8)other students . (normally 8)
Also initial self-assess stage (reflection) … used Also initial self-assess stage (reflection) … used as a as a standard of expectationstandard of expectation
Internet, not Web-based system (developed in Internet, not Web-based system (developed in Visual Basic / Access)Visual Basic / Access)
![Page 5: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/5.jpg)
55
![Page 6: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/6.jpg)
66
Having done the marking, what Having done the marking, what next?next?
Students should receive feedbackStudents should receive feedback
What feedback?What feedback?– Marks Marks – CommentsComments
Which is most important?Which is most important?– To students or staffTo students or staff
![Page 7: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/7.jpg)
77
![Page 8: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/8.jpg)
88
AUTOMATICALLY EMAIL THE MARKER ..
ANONYMOUS
![Page 9: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/9.jpg)
99
What should the marker do?What should the marker do?ReflectReflect
Look at essay againLook at essay again Take into account the essay owner’s Take into account the essay owner’s
commentscomments Further clarification (if it is needed, then is Further clarification (if it is needed, then is
this a ‘black mark’ against the marker?)this a ‘black mark’ against the marker?) Try to ‘appease’ the essay owner?Try to ‘appease’ the essay owner? Modify mark based upon reflection?Modify mark based upon reflection? Give more feedbackGive more feedback
![Page 10: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/10.jpg)
1010
![Page 11: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/11.jpg)
1111
Must be rewarded for doing the Must be rewarded for doing the ‘mark for marking’ process .. Based ‘mark for marking’ process .. Based
on qualityon quality How to judge?How to judge?
Standard of expectation (self-assessment)Standard of expectation (self-assessment) Marking consistencyMarking consistency Commenting, quality, measure against markCommenting, quality, measure against mark Discussion ElementDiscussion Element Need for additional comments – black mark?Need for additional comments – black mark? Reaction to requests / further clarificationReaction to requests / further clarification
![Page 12: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/12.jpg)
1212
Standard of ExpectationStandard of ExpectationSelf-Assess = 52%Self-Assess = 52%
Peer-Assessed as = 58%Peer-Assessed as = 58%ESSAYESSAY Compensated Compensated
Median MarkMedian MarkStudent Student
MarkMarkDifferenceDifference
(absolute)(absolute)
WW 65%65% 57%57% -8% (2)-8% (2)
XX 48%48% 36%36% -12% (6)-12% (6)
YY 53%53% 49%49% -4% (2)-4% (2)
ZZ 61%61% 59%59% -2% (4)-2% (4)
-6.5% (3.5)-6.5% (3.5)
![Page 13: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/13.jpg)
1313
![Page 14: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/14.jpg)
1414
How easy to get mark for marking?How easy to get mark for marking?
Statistically fairly easy to create a mark for Statistically fairly easy to create a mark for marking based upon marksmarking based upon marks
Take into account high and low markersTake into account high and low markers Standard of expectationStandard of expectation Consistency … judge against final mark Consistency … judge against final mark
awarded for an essay (compensated awarded for an essay (compensated median)median)
What about the comments?What about the comments?
![Page 15: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/15.jpg)
1515
Feedback IndexFeedback Index
Produce an index that reflects the quality of Produce an index that reflects the quality of commentingcommenting
Produce an average feedback index for an Produce an average feedback index for an essayessay
Compare against marker in a similar manner Compare against marker in a similar manner to marks analysisto marks analysis
Where does this feedback index come from Where does this feedback index come from and is it valid?and is it valid?
![Page 16: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/16.jpg)
1616
The way to get the feedback index?The way to get the feedback index?
Develop an application??Develop an application??– C-Rater?C-Rater?
Spelling MistakesSpelling Mistakes Similar Meanings?Similar Meanings?
– That was coolThat was cool– Really ChocReally Choc– Really Good EssayReally Good Essay
ManuallyManually
![Page 17: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/17.jpg)
1717
![Page 18: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/18.jpg)
1818
Commonality!!Commonality!!
In the 67 essays that were markedIn the 67 essays that were marked– Only 96 commentsOnly 96 comments– 44% positive and 56% negative44% positive and 56% negative– Highly criticalHighly critical if something not explained properly (21% of total if something not explained properly (21% of total
comments (of which 73% were negative)comments (of which 73% were negative) Better students were more critical than weaker students .. Better students were more critical than weaker students ..
– Better understanding permitting criticism?Better understanding permitting criticism?– Confidence?Confidence?– Hostility?Hostility?
Comments grouped into 10 categoriesComments grouped into 10 categories Need to QUANTIFY these comments .. Feedback indexNeed to QUANTIFY these comments .. Feedback index Create a database holding positive & negative (by Create a database holding positive & negative (by
category)category)
![Page 19: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/19.jpg)
1919
![Page 20: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/20.jpg)
2020
![Page 21: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/21.jpg)
2121
![Page 22: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/22.jpg)
2222
![Page 23: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/23.jpg)
2323
![Page 24: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/24.jpg)
2424
![Page 25: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/25.jpg)
2525
Time Consuming?Time Consuming?
Can we formulate the marking processCan we formulate the marking process
Take away need for quantification process Take away need for quantification process of analyzing commentsof analyzing comments
Is it still peer-assessment if the students are Is it still peer-assessment if the students are told what to say?told what to say?
![Page 26: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/26.jpg)
2626
![Page 27: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/27.jpg)
2727
STUDENT FRED
REFERENCES: Positive ………
Negative …….
Personal Valuation
5, 3, 2, 1
3, 1, 2
![Page 28: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/28.jpg)
2828
Some points outstandingSome points outstanding What should students do if they identify What should students do if they identify
plagiarism?plagiarism? Is it ethical to get students to mark the work of Is it ethical to get students to mark the work of
their peers?their peers? Is a computerised solution valid for all?Is a computerised solution valid for all? At what age / level can we trust the use of peer At what age / level can we trust the use of peer
assessment?assessment? How do we assess the time required to perform How do we assess the time required to perform
the marking task?the marking task?
![Page 29: Phil Davies School of Computing University of Glamorgan “Super U”](https://reader035.vdocuments.mx/reader035/viewer/2022062806/56814f8b550346895dbd451d/html5/thumbnails/29.jpg)
2929
Peer-Assessment: Peer-Assessment: No marks required, just No marks required, just
feedback?feedback?
Not there yetNot there yet Feedback index results are very positiveFeedback index results are very positive Require more evaluation before totally automated Require more evaluation before totally automated
system is possiblesystem is possible Getting there .. Removing subjectivity in markingGetting there .. Removing subjectivity in marking