11
Phil DaviesPhil DaviesSchool of ComputingSchool of Computing
University of Glamorgan “Super U”University of Glamorgan “Super U”
Peer-Assessment:Peer-Assessment: No marks required, just No marks required, just
feedback?feedback?Evaluating the Quality of Evaluating the Quality of
Computerized Peer-Feedback Computerized Peer-Feedback compared with Computerized compared with Computerized
Peer-Marking Peer-Marking
22
Definition of Peer-Assessment?Definition of Peer-Assessment?
In describing the teacher ..In describing the teacher ..
A tall b******, so he was. A tall thin, mean b******, A tall b******, so he was. A tall thin, mean b******, with a baldy head like a lightbulb. He’d make us with a baldy head like a lightbulb. He’d make us mark each other’s work, then for every wrong mark each other’s work, then for every wrong mark we got, we’d get a thump. That way – he mark we got, we’d get a thump. That way – he paused – ‘we were implicated in each other’s pain’paused – ‘we were implicated in each other’s pain’
McCarthy’s Bar McCarthy’s Bar (Pete McCarthy, 2000,page 68)(Pete McCarthy, 2000,page 68)
33
Why Peer-AssessmentWhy Peer-Assessment Community of Practice!!Community of Practice!! Working together to learn (and be assessed)Working together to learn (and be assessed) PerceptionsPerceptions
– Staff > Lecturer getting easy life not markingStaff > Lecturer getting easy life not marking– Student> Lecturer getting easy life not markingStudent> Lecturer getting easy life not marking
Student Awareness of Benefits …. IMPORTANT Student Awareness of Benefits …. IMPORTANT TANGIBILITYTANGIBILITY
Success dependant upon scalability Success dependant upon scalability (computerisation) .. “Student Numbers have risen (computerisation) .. “Student Numbers have risen dramatically since 1991 without a concomitant dramatically since 1991 without a concomitant increase in resources” (Pond et al, 1995)increase in resources” (Pond et al, 1995)
44
Computerised Peer-AssessmentComputerised Peer-Assessment
CAP SystemCAP System
Permits students to mark & comment the work of Permits students to mark & comment the work of other students . (normally 8)other students . (normally 8)
Also initial self-assess stage (reflection) … used Also initial self-assess stage (reflection) … used as a as a standard of expectationstandard of expectation
Internet, not Web-based system (developed in Internet, not Web-based system (developed in Visual Basic / Access)Visual Basic / Access)
55
66
Having done the marking, what Having done the marking, what next?next?
Students should receive feedbackStudents should receive feedback
What feedback?What feedback?– Marks Marks – CommentsComments
Which is most important?Which is most important?– To students or staffTo students or staff
77
88
AUTOMATICALLY EMAIL THE MARKER ..
ANONYMOUS
99
What should the marker do?What should the marker do?ReflectReflect
Look at essay againLook at essay again Take into account the essay owner’s Take into account the essay owner’s
commentscomments Further clarification (if it is needed, then is Further clarification (if it is needed, then is
this a ‘black mark’ against the marker?)this a ‘black mark’ against the marker?) Try to ‘appease’ the essay owner?Try to ‘appease’ the essay owner? Modify mark based upon reflection?Modify mark based upon reflection? Give more feedbackGive more feedback
1010
1111
Must be rewarded for doing the Must be rewarded for doing the ‘mark for marking’ process .. Based ‘mark for marking’ process .. Based
on qualityon quality How to judge?How to judge?
Standard of expectation (self-assessment)Standard of expectation (self-assessment) Marking consistencyMarking consistency Commenting, quality, measure against markCommenting, quality, measure against mark Discussion ElementDiscussion Element Need for additional comments – black mark?Need for additional comments – black mark? Reaction to requests / further clarificationReaction to requests / further clarification
1212
Standard of ExpectationStandard of ExpectationSelf-Assess = 52%Self-Assess = 52%
Peer-Assessed as = 58%Peer-Assessed as = 58%ESSAYESSAY Compensated Compensated
Median MarkMedian MarkStudent Student
MarkMarkDifferenceDifference
(absolute)(absolute)
WW 65%65% 57%57% -8% (2)-8% (2)
XX 48%48% 36%36% -12% (6)-12% (6)
YY 53%53% 49%49% -4% (2)-4% (2)
ZZ 61%61% 59%59% -2% (4)-2% (4)
-6.5% (3.5)-6.5% (3.5)
1313
1414
How easy to get mark for marking?How easy to get mark for marking?
Statistically fairly easy to create a mark for Statistically fairly easy to create a mark for marking based upon marksmarking based upon marks
Take into account high and low markersTake into account high and low markers Standard of expectationStandard of expectation Consistency … judge against final mark Consistency … judge against final mark
awarded for an essay (compensated awarded for an essay (compensated median)median)
What about the comments?What about the comments?
1515
Feedback IndexFeedback Index
Produce an index that reflects the quality of Produce an index that reflects the quality of commentingcommenting
Produce an average feedback index for an Produce an average feedback index for an essayessay
Compare against marker in a similar manner Compare against marker in a similar manner to marks analysisto marks analysis
Where does this feedback index come from Where does this feedback index come from and is it valid?and is it valid?
1616
The way to get the feedback index?The way to get the feedback index?
Develop an application??Develop an application??– C-Rater?C-Rater?
Spelling MistakesSpelling Mistakes Similar Meanings?Similar Meanings?
– That was coolThat was cool– Really ChocReally Choc– Really Good EssayReally Good Essay
ManuallyManually
1717
1818
Commonality!!Commonality!!
In the 67 essays that were markedIn the 67 essays that were marked– Only 96 commentsOnly 96 comments– 44% positive and 56% negative44% positive and 56% negative– Highly criticalHighly critical if something not explained properly (21% of total if something not explained properly (21% of total
comments (of which 73% were negative)comments (of which 73% were negative) Better students were more critical than weaker students .. Better students were more critical than weaker students ..
– Better understanding permitting criticism?Better understanding permitting criticism?– Confidence?Confidence?– Hostility?Hostility?
Comments grouped into 10 categoriesComments grouped into 10 categories Need to QUANTIFY these comments .. Feedback indexNeed to QUANTIFY these comments .. Feedback index Create a database holding positive & negative (by Create a database holding positive & negative (by
category)category)
1919
2020
2121
2222
2323
2424
2525
Time Consuming?Time Consuming?
Can we formulate the marking processCan we formulate the marking process
Take away need for quantification process Take away need for quantification process of analyzing commentsof analyzing comments
Is it still peer-assessment if the students are Is it still peer-assessment if the students are told what to say?told what to say?
2626
2727
STUDENT FRED
REFERENCES: Positive ………
Negative …….
Personal Valuation
5, 3, 2, 1
3, 1, 2
2828
Some points outstandingSome points outstanding What should students do if they identify What should students do if they identify
plagiarism?plagiarism? Is it ethical to get students to mark the work of Is it ethical to get students to mark the work of
their peers?their peers? Is a computerised solution valid for all?Is a computerised solution valid for all? At what age / level can we trust the use of peer At what age / level can we trust the use of peer
assessment?assessment? How do we assess the time required to perform How do we assess the time required to perform
the marking task?the marking task?
2929
Peer-Assessment: Peer-Assessment: No marks required, just No marks required, just
feedback?feedback?
Not there yetNot there yet Feedback index results are very positiveFeedback index results are very positive Require more evaluation before totally automated Require more evaluation before totally automated
system is possiblesystem is possible Getting there .. Removing subjectivity in markingGetting there .. Removing subjectivity in marking