1 Assessment Design

xxxxxxxyyyyyyy

Assessment workshops

This session is one of a series on assessment:1. Assessment Design 2. Marking 3. Feedback:

Purpose

To help you to design your assessment... …in line with the expectations of a British

university.

Papers

You were asked to bring with you a Module Spec and examples of your assessment methods for this module

We’ll use these during the session.

Assessment design

Two aspects:1. Designing the system of assessment ~

the responsibility of the university2. Designing the assessment undertaken by

the students ~ your responsibility Session mainly about 2, but...

Key resources

Programme and Module Specifications

Assessment Guidelines

Coursework Submission, Marking and Feedback Policy

Policy for Control Procedures for Examination and Assessment Marking and Recording of Marks

System of assessment

Assessment approach specified Papers aligned with ILOs Papers checked internally Papers commented on by external examiner Responses to comments recorded (or the

loop is not closed) Papers double-marked to ensure fairness Marks reviewed by external examiner for

standards and consistency.

Why do we assess students?

Classify or grade students Enable student progression Guide improvement Indicate what the module is about Inform students’ choice of options Provide feedback on our teaching Motivate students Provide statistics for the programme/course Provide evidence for quality purposes Etc.

Why do we assess? 2

Summative ~ summary measure of performance

Formative ~ feedback to help students improve

Informative ~ guidance on what is expected.

Issues in assessment

Summative Standards, consistency, security, etc Balance of exam and coursework Move to profiles to provide more information

Formative Timeliness, helpfulness

Informative Consistency between teaching and assessment

More on these later.

Issues in assessment 2

The amount of assessment is a key issue Do we over-assess our students? Too much assessment wastes the students’

time and reduces their motivation to learn It also wastes our time and reduces our

motivation! How much is enough? Assessment must be driven by the ILOs Each ILO must be both assessable and

assessed... ...but duplication is wasteful.

Module assessment matrix

Assessment methods must suit the ILO Can surveying skills be assessed entirely on

paper?

Method

ILO

Method 1

Method 2

Method 3

Method 4

ILO 1 ILO 2 ILO 3 ILO 4 ILO 5

Your assessment matrix

Individually please spend a few minutes creating part of the assessment matrix for your module

Show how you assess one or two ILOs, preferably using different assessment methods.

Sharing...

...your assessment matrices In twos or threes please share information

on your ILOs and your assessment methods What methods have been more or less

successful? What imaginative assessment methods

emerged? What time-saving methods of assessment

emerged? etc

Plenary

Programme assessment matrix

Assessment methods must suit the programme ILOs too.

Module

ILO

Module 1

Module 2

Module 3

Module 4

ILO 1 ILO 2 ILO 3 ILO 4 ILO 5

Subject Advisors’ concerns

What follows is a list of some of the concerns about assessment expressed by the Subject Advisors

Others will be picked up in the session on feedback

But setting and marking are very closely linked so there will be some duplication

Our task is to offer recommendations for ways of addressing these.

SAs’ assessment concerns 1

Disparity between coursework averages and exam averages: The relatively high number of students

passing a module chiefly on high coursework marks (but with low/fail exam marks)

Example recommendations Set more demanding coursework Mark coursework more rigorously Tell students more about the form of the

exam Let them practise answering exam

questions.

SAs’ assessment concerns 2

Mark distributions: Some coursework averages impossibly high Large number of students achieving a grade D …large numbers of students are failing

modules The failure rate of some modules…..is

alarming and wholly unacceptable… Student performance is atypical with regard to

distribution of marks…..they are not satisfactory at this time

Example recommendations Same coursework recommendation as 1 Raise intake standards Provide more support to students.

SAs’ assessment concerns 3

Documentation: Confusing assessment statement Module specifications do not state the

actual number of assessments What the student was required to complete/

submit was not always evident …..making it impossible to assess the standard of the student’s work

Example recommendations Tidy up documentation. Ensure clarity of expression – all assessment briefs/exams to

be checked for clarity. And Faculties and Departments need to have a structure to ensure that this is monitored.

SAs’ assessment concerns 4

Weaknesses in assessment strategies: Over assessment Repetitive assessment of the same ILO Inappropriate assessment types Failure to deliver the module’s stated

assessment strategy Over concentration on coursework can

lead to poor exam preparation Some coursework set at too easy a level

without proper reference to ILOs A bias on behalf of students towards

assessed work of any sort and a particular reliance upon it to pass courses

SAs’ assessment concerns 4

Weaknesses in assessment strategies: Over assessment Repetitive assessment of the same ILO Inappropriate assessment types Failure to deliver the module’s stated

assessment strategy Over concentration on coursework can lead

to poor exam preparation..... Example recommendations

Use the assessment matrix Do the assessment arithmetic with students Set more demanding coursework.

SAs’ assessment concerns 5

Marking of group projects/assignments: Clarity required on award of marks to

individual students undertaking group tasks

Example recommendations Agree and share marking scheme.

SAs’ assessment concerns 6

Award of marks: …clear evidence of generous/lenient marking

Grade descriptors are not always observed …no indication how marks are being awarded

for presentations Generous/lenient exam marking Coursework where many students achieve

over 80% are not appropriate. 20% of the marks should be reserved for excellence beyond the normal excellence.

Example recommendations Set more demanding assessment and mark to

it.

SAs’ assessment concerns 7

Scrutiny of marks: …develop and refine the mechanism for

monitoring and mediating module marks before they are presented to the board

Adopt a robust practice of double marking since most examination scripts did not appear to have been subjected to a robust procedure of second/double marking

Example recommendations Develop and use processes for

moderation of marks.

Mark distributions

Many concerns about too many low marks and too many high marks

Different causes, different solutions Low marks = weak students needing more

support Too many high marks = easy assessments

and generous marking Toughen up the assessment by thinking

about the assessment hierarchy Ensure appropriate moderation of marks.

After Bloom

Create

Evaluate

Analyse

Apply

Understand

Remember

Blooming obvious

The good students will be able to operate at these higher levels

The weaker students will not

So we should ask some of our questions at this level…

…with more of these higher level questions in the later years.

Learning verbs (Hall, 1976)

Define Describe Identify List Match Name Order Recognize Recall

Appraise Argue Assess Compare Contrast Defend Discriminate Evaluate Explain Justify Predict Summarize

Words on the left are at lower levels in the hierarchy

Assessment assumptions

Students can fail The good students

should have the chance to show that they are good

There aren’t any trick questions

You mark what you see, not what you hope to see

One person doesn’t decide their mark

You should mark using the full range

Students know how they are going to be assessed

The assessment criteria are clear to all

Abilities – and marks – are distributed reasonably normally

“reasonably normally”

Resolutions

What are you going to do this semester to help address the assessment concerns?

2 Marking

xxxxxxxyyyyyyy

Assessment workshops

This session is one of a series on assessment:1. Assessment Design 2. Marking 3. Feedback

Purpose

To help you to mark your assessment... …in line with the expectations of a British

university.

Key resources Programme and Module Specifications

Assessment Guidelines

Coursework Submission, Marking and Feedback Policy

Policy for Control Procedures for Examination and Assessment Marking and Recording of Marks

Coursework Submission and Statement of Academic Honesty Form

Feedback Form

Papers

You were asked to bring with you examples of a range of assessments that you have used or will use

We’ll work with these during the session.

Subject Advisors’ concerns

Remember some of the Subject Advisors’ concerns about marking...

SAs’ assessment concerns 2

Mark distributions: Some coursework averages impossibly high Large number of students achieving a grade D …large numbers of students are failing

modules The failure rate of some modules…..is

alarming and wholly unacceptable… Student performance is atypical with regard to

distribution of marks…..they are not satisfactory at this time

Example recommendations Same coursework recommendation as 1 Raise intake standards Provide more support to students.

SAs’ assessment concerns 6

Award of marks: …clear evidence of generous/lenient

marking Grade descriptors are not always observed

…no indication how marks are being awarded for presentations

Generous/lenient exam marking Coursework where many students achieve

over 80% are not appropriate. 20% of the marks should be reserved for excellence beyond the normal excellence.

SAs’ assessment concerns 6

Example recommendations Set more demanding assessment and mark to it.

Marking is not to be impressionistic, nor according to the Egyptian scale which gives much higher marks, but according to the B(ritish)UE scale which is in %s.

All marking needs to be done, and justified according to the official marking criteria. These are customised from the generic University criteria and will vary for P, I and H levels.

SAs’ assessment concerns 6

Example recommendations (continued)

Double marking is best when it is blind. It should not be a matter of just sharing the paper to be marked with one's office mate since this can boil down to superficial approval if not virtual collusion. It should be a real re-marking.

Marks are not finally given by the module leader but the Module/Programme Exam Boards, where they may be moderated. What are the benefits of this system? At any rate, it means that teachers should not dispense final marks to students.

Pre-requisites

Your assessment must be designed in a way that will allow your best students to show their strengths

If you set assessments that are at the bottom of the assessment hierarchy then it will be hard for the best students to show that they are better.

Sharing

In twos or threes please compare some of your assessments and try to position them on the assessment hierarchy.

Up a level or two

Now take one or two questions from within your groups and revise them so that they move up a level or two.

What are you looking for?

The key is to be clear about what the different levels of marks mean

A good way of thinking about this is to be able to respond to these questions from students... “What will get me a mark of 90%?” “What will get me a mark of 70%?” “What will get me a mark of 50%?” “What will get me a mark of 30%?”

How would you answer them? Try this in twos or threes - and see if you are

convinced by what you are told.

Marking exercise

We’re now going to take a short answer from your broad subject area and mark it individually

But first, an introduction to the question and the target audience (you will expect more from final year students than from prep year students).

This needs a senior person from the discipline who is prepared to offer their mark and their comments.

Context

Question Target audience Comments ...and a reminder that you are looking for

the Rifle Approach not the Shotgun Approach.

Shotgun Rifle

Rifle not shotgun!

Marking

Now mark the piece of work We’re looking for a mark and one or two

sentences that justify the mark that you have awarded.

Marks

90+

80-89

70-79

60-69

50-59

40-49

30-39

20-29

10-19

<10

Discussion

In your twos and threes please compare and contrast your marks and your justifications.

The mark!

And now the mark that the setter gave, with their justification.

Resolutions

What are you going to do this semester to help address the marking concerns?

3 Feedback

xxxxxxxyyyyyyy

Assessment workshops

This session is one of a series on assessment:1. Assessment Design 2. Marking 3. Feedback

Purpose

To help you to provide feedback... …in line with the expectations of a British

university.

Key resources Programme and Module Specifications

Assessment Guidelines

Coursework Submission, Marking and Feedback Policy

Policy for Control Procedures for Examination and Assessment Marking and Recording of Marks

Coursework Submission and Statement of Academic Honesty Form

Feedback Form

Papers

You were asked to bring with you an example of a piece of work on which you have offered some assessment comments

We’ll work with this during the session.

Feedback

Performance without feedback is a missed opportunity

People need feedback if they are to improve Feedback is formative!

Battleships?

A mark tells them something, but it discourages reflection

“This assignment handled xxxx and yyyy well, but it would have been much better if...”

Important to tell the students how they can improve, or how will they know?

It can be a game of battleships.

Performance and marks

In pairs, please sit facing each other Place a piece of A4 paper on a pad or book

in front of A B closes his or her eyes A draws a cross somewhere near the middle

of the paper Keeping their eyes closed, B puts a mark

with their pen on the paper 5 times, trying to hit the cross

After each attempt A gives B a mark for the attempt.

Feedback

From the As What happened to performance?

From the Bs How did that feel?

Performance and feedback

In pairs, please sit facing each other Place a piece of A4 paper on a pad or book

in front of B A closes his or her eyes B draws a cross somewhere near the middle

of the paper Keeping their eyes closed, A puts a mark

with their pen on the paper 5 times, trying to hit the cross

After each attempt B gives A constructive verbal feedback on the attempt.

Feedback

From the Bs What happened to performance?

From the As How did that feel?

Your feedback

Please share your feedback examples and consider these questions...

Did your feedback help improve performance?

How would the students receiving this feedback feel?

SAs’ assessment concerns 8

Amount of feedback: Feedback is variable….no minimum

requirement/shared expectation among staff as to what is appropriate

…improve the level of feedback/evidence of marking particularly on group project assignments

…oral feedback is given…this should be backed up by some written examples

….a lack of evidence/examples of staff placing Generic Exam Feedback on eLearning despite this being stated in the Module Specification

SAs’ assessment concerns 9

Amount of feedback (continued): Staff argued that informal oral feedback

is given….but there appears to be no traceability for such practice

Over reliance on pre-prepared sheets (feedback)

Students’ use of feedback ….staff need to ‘educate’ students into

making proper or at least better use of written feedback.

SAs’ assessment concerns 10

Quality of feedback: …comments were rather terse and

tended to focus on the negative rather than the positive and negative aspects

….(feedback is) often late and often verbal only

Too many examples of coursework with simply a mark appended

A standard feedback form is used but this often appeared to be record grades/marks with no or extremely curtailed informative comments

SAs’ assessment concerns 11

Quality of feedback (continued): …improve and then maintain the level of

feedback given by TAs on the Coursework Submission and Feedback Form. Perhaps a check should be done by the Module Leader.

Language

What can we say about the sort of language that makes feedback more helpful?

Helpful feedback?

What are the characteristics of helpful feedback?

Please work in small groups to come up with a list of key points.

Helpful feedback

Evidence-based Timely Constructive Motivating Mixed – positive and negative Owned by the giver Personal to the receiver The Teaching and Learning Committee and

Senate are considering what feedback should be given by all staff. But it is only a minimum. 

Feedback sandwich

Positive

Positive

Negative

Positive and negative needed

Although solitary under normal prevailing circumstances, raccoons may congregate simultaneously in certain situations of artificially enhanced nutrient resource availability.

Subject Advisors’ concerns

The Subject Advisors offer quite a lot of feedback on feedback

Here it is...

SAs’ assessment concerns 8

Amount of feedback: Feedback is variable….no minimum

requirement/shared expectation among staff as to what is appropriate

…improve the level of feedback/evidence of marking particularly on group project assignments

…oral feedback is given…this should be backed up by some written examples

….a lack of evidence/examples of staff placing Generic Exam Feedback on eLearning despite this being stated in the Module Specification

SAs’ assessment concerns 9

Amount of feedback (continued): Staff argued that informal oral feedback

is given….but there appears to be no traceability for such practice

Over reliance on pre-prepared sheets (feedback)

Students’ use of feedback ….staff need to ‘educate’ students into

making proper or at least better use of written feedback.

SAs’ assessment concerns 10

Quality of feedback: …comments were rather terse and

tended to focus on the negative rather than the positive and negative aspects

….(feedback is) often late and often verbal only

Too many examples of coursework with simply a mark appended

A standard feedback form is used but this often appeared to be record grades/marks with no or extremely curtailed informative comments

SAs’ assessment concerns 11

Quality of feedback (continued): …improve and then maintain the level of

feedback given by TAs on the Coursework Submission and Feedback Form. Perhaps a check should be done by the Module Leader.

Key concepts

We need to develop a culture where giving and receiving feedback is seen as normal

We need light-touch systems that check that feedback is being given

We need to develop the skills of giving effective feedback

We need to get students to expect to get feedback and to use it!

Feedback practice

Please write some feedback on this first year essay...

Feedback comparison

Now swap your feedback with someone and read their version

Would it help you to improve your performance?

How did you feel as you read it? Give each other feedback on your feedback.

Resolutions

What are you going to do this semester to help address the feedback concerns?