TMA feedback: can we do better?

Mirabelle Walker

Centre for Open Learning of Mathematics, Science Computing and Technology (COLMSCT)

FAST project findings (1)

• Quantity and timing of feedback– Sufficient feedback is provided, often enough and in enough detail– The feedback is provided quickly enough to be useful to students

• Quality of feedback– Feedback focuses on students’ performance and learning, and on actions

under their control– Feedback is appropriate to the purpose of the assignment and to its criteria

for success– Feedback is appropriate in relation to students’ understanding of what they

are supposed to be doing

• Student response to feedback– Feedback is received and attended to– Feedback is acted upon by the student

Gibbs & Simpson (2004–5)

FAST project findings (2)

What students said about their TMA feedback• Received plenty of it• Motivated by praise and encouragement• Mainly received within three weeks• Read feedback but rarely acted on it

FAST project findings (3)

Category of comments

0102030405060

Per

cen

tag

e

FAST project findings (4)

Type of content comment

0

10

20

30

40

50

60

70

errors omissions irrelevancies clarifications

Per

cen

tag

e

Idea of ‘depth’ of comment

• Depth 1 – acknowledges e.g. ‘more needed here’; ‘good’

• Depth 2 – corrects / amplifies e.g. ‘you needed to mention xxxx’; ‘a good introduction’

• Depth 3 – explains e.g. ‘you needed to mention xxxx because …’; ‘a good introduction because …’

FAST project findings (5)

Depth of comments

0

10

20

30

40

50

60

70

depth 1 depth 2 depth 3

Per

cen

tag

e

FAST project findings (6)

Significant weaknesses in current practice:• Too much emphasis on justifying the grade• Lack of shared understanding of assessment criteria

(students & CT)• ALs good at articulating students’ weaknesses;

explaining strengths problematic• Lack of holistic assessment of students’ work

My project

• Replicate some of FAST investigations in Technology – results similar?– Courses chosen: T173, T209 & T224– Analysis of feedback on sample TMAs– Telephone interviews with students

• Follow-up action with ALs, monitors, CTs, STs as appropriate – and review

Progress so far

• Feedback analysed on all three courses• Some results circulated to T209 & T224 ALs• Some comparisons with Science made• Telephone interviews conducted on T209 & T224 –

T173 next month• Some analysis of telephone feedback done – awaiting

T173 to finalise

Category percentages compared – Technology & Science

0

10

20

30

40

50

60

Tech

Science

Category percentages compared – Technology courses & Science

0

10

20

30

40

50

60

Perc

en

tag

e T173

T209

T224

Science

Category percentages compared – Technology (not T209) & Science

0

10

20

30

40

50

60

T173 & T224

Science

Type of content comment compared – Technology & Science

0

10

20

30

40

50

60

70

errors omissions irrelevancies clarifications

Per

cen

tag

e

Tech

Science

Type of content comment compared – Technology courses & Science

0

10

20

30

40

50

60

70

errors omissions irrelevancies clarifications

Per

cen

tag

e T173

T209

T224

Science

Type of content commentcompared within T173

T173 Questions 1 & 4

0

0.5

1

1.5

2

2.5

errors omissions irrelevancies clarifications

no

. p

er s

tud

ent

Qn1

Qn 4

Depth percentages compared – Technology & Science

0

10

20

30

40

50

60

70

80

Tech Science

Per

cen

atg

e depth 1

depth 2

depth 3

Depth percentages compared – Technology courses & Science

0

10

20

30

40

50

60

70

80

90

T173 T209 T224 Science

Per

cen

tag

e depth 1

depth 2

depth 3

Further comparison of depths

0%

20%

40%

60%

80%

100%

Per

cen

tag

e

depth 3

depth 2

depth 1

Number of comments per student

Mean Median Lowest Highest

T173 29 26 4 56

T209 36 35 9 (twice) 81

T224 17 15 8 (twice) 48

Science 39

Action taken on T209 and T224

• Document ‘Using TMA comments to good effect’ prepared for each course– Explains ideas of ‘depth’ and ‘feed forward’– Contains course-specific examples of depth 2 & depth

3 comments• Sent out in first tutor mailing (2006) with commendation

from Course Chair• Monitors briefed accordingly

Telephone surveys

• T209 and T224 complete – carried out immediately after end of course

• Examined– student’s perception of usefulness/helpfulness of

feedback– whether (& how) student had used the feedback in

any future TMA / the ECA / the exam– student’s preferences regarding placing of comments:

on PT3, script, (pro-forma)

Preliminary findings (1)

• Students are eager to receive their marked TMAs and do read the feedback …

• … but they do not necessarily use the feedback again in the rest of the course (approx 20% said they never used it)

• T209 students were likely to make more use of the feedback later in the course than T224 students

• T209 students particularly mentioned using skills development feedback

Preliminary findings (2)

• Not all students found that tackling the TMA, and the feedback they subsequently received, encouraged them to study the rest of the course

• A small number of students said they were disappointed with the quality of the feedback (but some were surprisingly accepting)

Preliminary findings (3)

• Overwhelming majority of students value comments on the script the most:– ‘where lost marks made clear’– ‘tells me exactly where the mistake is’– ‘very specific’– ‘more evidenced against actual text’– ‘easier seeing my work with comments relating to it’

Aside about feedback and eTMAs

• We can’t assume that students read the PT3 first – or even at all

• We can’t assume that students find and read a separate marking document sent back with the marked TMA

• Some (most?) students find juggling documents on the screen awkward/difficult

• It’s easy for tutors to place comments exactly where they apply

• Turn-around times longer: symptom of a problem?

Action taken on T209 and T224

• T209 and T224:Students reminded to look for and read the PT3

• T209:Separate pro-forma dropped for eTMAs; tutors asked to copy and paste grids at end of questions

FAST project findings in Technology?

Significant weaknesses in current practice:

• Too much emphasis on justifying the grade

• Lack of shared understanding of assessment criteria (students & CT) varied

• ALs good at articulating students’ weaknesses; explaining strengths problematic

• Lack of holistic assessment of students’ work

Good feedback in Technology?

• Quantity and timing of feedback– Sufficient feedback is provided, often enough and in enough detail In

general– The feedback is provided quickly enough to be useful to students In general

• Quality of feedback– Feedback focuses on students’ performance and learning Too much biased

towards performance on this TMA– Feedback is appropriate to the purpose of the assignment and to its criteria

Purpose and criteria often implicit– Feedback is appropriate in relation to students’ understanding of what they

are supposed to be doing Sometimes, but more explanations would be helpful

• Student response to feedback– Feedback is received by students, and attended to Yes– Feedback is acted upon by the student Not sufficiently

Implications for Course Teams

• Different sorts of questions and criteria elicit different types of feedback

• (or) To elicit particular feedback, write the question and criteria accordingly – maybe even write the course material accordingly

More implications for CTs

• Be explicit with ourselves and with ALs and students about what an assignment’s purpose and criteria are

• Don’t assume that ALs will instinctively know what sort of feedback we’re hoping for – be explicit in the marking guide (or elsewhere)

Implications for Associate Lecturers

• Need for shift towards emphasis on supporting student’s learning and progress through course, rather than just explaining what was wrong in this particular TMA

• That implies more student-centred feedback – and more holistic feedback on PT3s

• It may also imply giving as much (more?) emphasis to feedback as to marks

Implications for monitors

• Need to shift emphasis from ‘Was the mark OK?’ towards ‘Was the feedback OK?’

• Need to encourage appropriate forms of feedback (and discourage non-appropriate ones?)

References

• FAST presentations given at Open University, 10 February 2005

• Gibbs, G & Simpson, C (2004–5) ‘Conditions under which assessment supports students’ learning’ Learning and teaching in higher education 1(1) pp 3–31; available via http://www.open.ac.uk/science/fdtl/pub.htm

Centre for Open Learning of Mathematics, Science Computing and Technology (COLMSCT) The Open UniversityWalton HallMilton KeynesMK7 6AA

http://cetl.open.ac.uk/colmsct