electronic marking of examinations pete thomas open university [email protected]
TRANSCRIPT
Electronic Marking of Examinations
Pete ThomasOpen University
[email protected]://mcs.open.ac.uk/eap
The Context
1. Distance education – Open University, UK
2. On-line study
3. Improving assessment
4. Enhancing the student experience
5. Computing
Distance Education: OU Style
Open University(HQ)
Marker
Marker
ExamCentre
ExamCentre
ExamCentre
ExamCentre
ExamCentre
Typical Computing course has 2 – 4 thousand students
Motivation
“The gap between the environment in which students learn and how they are assessed is widening … there is a radical discontinuity in their educational experience.”
Race et al. (1999)
Electronic Assessment Project Aims for Examinations
• To put the whole examination process on-line.• To design an appropriate exam paper.• To define the client and server functions.• To investigate invigilation issues.• To provide automatic marking of free text.• To provide useful feedback on answers.• To provide induction to the process.• To provide support for examiners.
Marking Exam Scripts
• The objects in the examination system:– Questions and sub-questions– Specimen (sample) solutions– Mark schemes– Rubrics (rules)– Student answers
• Marking consists of – Comparing an answer with a specimen solution and
assigning a mark– Applying the rubric
Issues
• Problems with questions– Incomplete, erroneous or inconsistent data– Asking for an inappropriate conclusion– Ambiguity
• Checking used to remove defects
• Errors come to light:– When candidates read the question– When examiners attempt to grade answers
Issues
• Problems with answers– Understanding the question– Poorly expressed ideas– Lack of knowledge– Poor language skills (spelling)– Use of abbreviations of own devising– Lack of typing skills
• Actions to address the problems– Standardization between markers
Issues
• Problems with specimen solutions– Do not match the question (as perceived by
candidates)– Numerous acceptable alternatives– Incomplete – Incorrect
• Actions to address problems– Expect markers to use professional
judgement– Standardization between markers
Issues
• Problems with mark schemes– Incorrect– Inappropriate for the question as perceived by
candidates– Inappropriate for the difficulty of the question
• Actions to address problems– Revise in the light of experience– Standardization between markers
Electronic Marking
• A specimen solution can consist of more than one solution.
• Each solution can have a different mark.
• A solution can be split into parts with each part having its own marks.
• In general, an ‘electronic solution’ can be composed of a set of alternatives and
each alternative is composed of parts.
Solution RepresentationSuppose that the solution to a (trivial) question consists of the following three phrases:
range memory locations, bounds registers,address outside range exception
AND (3, 6.0)
bounds registersrange memory locations address outside range exception
storage protection keys
storage protection keysbounds registers
OR (2, 1, 3.0)
1.51.5
3.0 3.0
Synonyms?
Thesaurus
First Experiment
20 student scripts from a conventional written exam
3 independent human markers
Question 7 12 (a) 12 (b) 12 (c) 12 (d) 12 Total Over all
Marks allocated
4 8 2 8 2 20 24
Marker averages
1.7 6.1 1.1 4.8 1.13 13.0 14.6
Tool averages
1.75 5.5 0.98 4.8 0.8 12.4 14.2
Second ExperimentElectronic mock examination (3 hours)
Full exam scripts (10 Part 1 and 3 Part 2 questions)
11 students
3 independent human markers
Automatic Score Markers’ mean score
Mean 44.18 56.25
St.Dev. 8.28 10.09
Automatic score is lower and there is a smaller standard deviation
Deficiencies
Description Number
Number of significant spelling errors in answers 8
Number of deficient solution trees (out of 39) 15
Number of thesaurus deficiencies (synonyms) 12
Number of deficient specimen solutions 2
Number of lexical deficiencies (abbreviations) 1
Number of language parsing errors (software errors) 2
Number of deficient questions 2
Deficiency Reduction• Examined one script taken at random and
corrected/amended the solution trees and the thesaurus.
• Repeated the process with 3 more scripts.
Automatic Marker Human Marker Mean
0 1 2 3 4
Mean 44.2 46.0 50.5 52.8 53.2 55.85
Comparison
35
40
45
50
55
60
65
70
40 45 50 55 60 65 70 75 80
Mark awarded by markers
Mar
k aw
ard
ed b
y el
ectr
on
ic t
oo
l
Correlations
Pearson correlation Part 1 Part 2 Total
Tutor 1 & Tutor 2 0.8260 0.9680 0.9456
Tutor 1 & Tutor 3 0.9531 0.9350 0.9281
Tutor 2 & Tutor 3 0.9279 0.9170 0.9362
All tutors & electronic 0.9026 0.8244 0.8604
Examiner Support
Thesaurus maintenance
Exam Building System
Further Experiments
Marker comparisons by part (%) April 03
0
10
20
30
40
50
60
70
Total Part I Part II
Au
R
N
Ad
M
Revised algorithm
Marker comparisons by part (%) April 03
0
10
20
30
40
50
60
70
Total Part I Part II
Au
R
N
Ad
M
Marking Diagrams (April 03)
• What features should a drawing tool provide?• How familiar should students be with the tool
prior to the examination?• How should the tool be provided to students in
order to be used under examination conditions?
• How should a diagram be represented for transmission to the server?
• How should a diagram be represented for grading purposes?
• How to grade a diagram?
Exam Question
Use the drawing tool to draw a diagram that illustrates how the data hazard inherent in the execution of the pair of instructions
ADD R2, R3, R1SUB R1, R5, R4
by a 4-stage pipeline, can be overcome.
Specimen Solution &Drawing Tool
Associations & Constraint Multiset Grammars
a box a link a box
the start box containing text
the end box containing text some text
Association → Link, Weight where (exists Box1 Box2 attached (Link.start, Box1.area) &attached (Link.end, Box2.area)
) {Association.from = Box1 &Association.to = Box2 &Association.text = Link.text &Association.weight = Weight
}
CMG for a pipelinePipeline → Association1, Association2, Association3
where (Association1.to = Association2.from &Association2.to = Association3.from &Association1.from.text.string = “fetch ADD” &Association2.from.text.string = “decode” &Association3.from.text.string = “execute” &Association3.to.text.string = “write R1” &
) {Pipeline.assoc1 = Association1 &Pipeline.assoc2 = Association2 &Pipeline.assoc3 = Association3 &
}
Initial Experiment
Student Human Markers (average)
Diagram Marking Tool
1 2.0 2.0
2 3.0 4.0
3 4.0 3.0
4 2.5 4.0
5 2.0 2.5
Mean 2.7 3.1
St. Dev 0.837 0.894
Thank [email protected]
http://mcs.open.ac.uk/eap