implementing ipsative assessment - university of london ... · pdf fileimplementing ipsative...

CENTRE FOR DISTANCE EDUCATION

TEACHING AND RESEARCH AWARDS ROUND 6

PROJECT REPORT

IMPLEMENTING IPSATIVE ASSESSMENT By Dr Gwyneth Hughes Kaori Okumoto Elizabeth Wood Faculty of Policy and Society Institute of Education 20 Bedford Way London WC1H 0AL Tel: 0207 911 5308 Email: [email protected]

October 2011

Implementing Ipsative Assessment

1

Implementing Ipsative assessment

A report on a Centre for Distance Education (CDE) funded study by Gwyneth Hughes, Kaori Okumoto and Elizabeth Wood

Summary

Distance learners rely on effective assessment and feedback for motivation and for learning. There is evidence from a previous study that an ipsative approach to assessment (assessment based on a comparison with a learner’s previous performance) might both motivate distance learners by developing self-awareness of progress and help them to take a longer-term approach to academic study. In this project an ipsative feedback scheme was developed with distance learning tutors and implemented using an ipsative feedback form for students to complete. These forms and subsequent tutor comments were evaluated and analysed for changes in feedback practice. Students and tutors were also interviewed about the value of the scheme. Findings indicated that ipsative feedback on progress has the potential to motivate learners and to encourage them to act on developmental feedback, but further intervention is needed to manage student expectations and to ensure that this intervention is sustainable and scalable.

1. Introduction

Distance learners are dependent on good feedback from tutors, yet assessment processes and feedback are often deemed unsatisfactory by students. Hughes (2011) has proposed that ipsative assessment (assessment based on a comparison with a previous performance rather than external criteria) may offer a solution. A previous CDE funded study explored the feedback that distance learners received (Hughes, et al. 2010) and found that there was very little evidence of ipsative feedback although learners would welcome more information about their progress. Tutors agreed that an ipsative approach which informs learners of how well they have implemented the recommendations from previous feedback would be helpful too, particularly for those students who are struggling with writing. Such feedback might also be especially motivating and supportive for learners who do not receive encouragement through high grades, but who nevertheless are progressing (Hughes, 2011). An ipsative approach requires careful tracking of a learner’s progress over several modules and could alert tutors to any students who were not acting on the feedback they have been given.

However, the preliminary study indicated that distance learning tutors do not have enough information about learner progression to provide learners with cumulative and ipsative feedback. An intervention was therefore planned to take the exploratory work further. This intervention included technological innovation with the current VLE to make all electronic feedback available to all tutors and the design of a new feedback submission form to explicitly address learner progress.

This report begins with a review of literature on assessment for distance learners and details of the rationale for the intervention. The report next describes how the intervention was evaluated using interviews and a tool to analyse written feedback. The main finding is that use of the form prompted some students to reflect on previous feedback and encouraged ipsative feedback from tutors that was motivational. However, there were further issues to be addressed and


2

recommendations arising from this intervention are presented along with suggestions for further work.

2. Literature Review: ipsative assessment and distance learning

Assessment has been receiving increasing attention in Higher Education with recognition that assessment drives learning as well as measuring outcomes. While much literature has addressed issues of reliability and validity of summative assessment practice (Bloxham et al.,Gibbs,2006), formative assessment has also received attention. Formative feedback aims to reduce the discrepancy between performance and desired goals by informing learners where they are now and where they need to go next (Hattie & Timperley, 2007; Sadler, 2005) and is especially important for the retention and motivation of distance learners (Simpson, forthcoming). However, there is plenty of evidence that formative feedback does not always perform this function effectively in higher education. Although students are usually interested in marks or grades, they might not be willing to engage with, or act upon, developmental feedback (feed forward). This may be because they have been de-motivated by feedback or because feed forward is not useable or it is unclear (Walker, 2009).

The impact of assessment on a learner’s sense of self-worth has a significant and often under-recognised influence on motivation. Comments and grades give individuals powerful messages about themselves as learners (Ivanic. et al.,2000; Stobart, 2008), and for many learners assessment produces negative emotions (Falchikov & Boud, 2007). Even when there are attempts to ‘sweeten’ negative or challenging feedback and results with positive comments, there are indications from other studies that many learners do not act on critical feedback (Crisp, 2007), either because they have focused only on the praise at a personal level, or because they have been overwhelmed by excessive critique (Lizzio & Wilson, 2008).

Handley, et al. (2008) reported that students were more likely to act upon feedback if they could see the immediate utility of it, for example, in helping with the next piece of work. But such task specific feed forward is only useful if the task is to be repeated in the short term and comments addressing task level criteria are not easy to apply to a new piece of work. Walker’s (2009) study from the UK Open University further supports a claim that developmental comments are more useful if they do not solely perform a short-term corrective function. Feedback comments on generic skills, such as thinking or writing skills, are most useable: these can be used for other assignments as well as improving on draft work and general statements can always be illustrated with points from the current assignment content. In the previous study, Hughes et al. (2010) also suggested that generic feedback illustrated by task specific examples is very helpful for distance learners.

To address problems with feedback clarity, Nicol & Macfarlane-Dick (2006) argue that feedback should be directed towards enabling learners to become self-regulating. Dialogue is the basis of a social constructivist rather than a transmissive view of learning and they suggest that discussion between teacher and learners or amongst peers about the criteria and standards can be very helpful and this is one of their key principles. However, individual face-to-face dialogue is difficult to implement in pressurised mass higher education and is not readily available for distance learners except by telephone or occasional meetings if tutors are pro-active (Simpson, 2008). Facilitating dialogue in mainly written communication is an alternative, but not easy to achieve. However, there is a different approach to assessment which might provide a solution: ipsative assessment and feedback.


3

Ipsative feedback to motivate and to encourage learner autonomy

Ipsative feedback has been proposed as a new approach that might address some of the current problems with formative assessment (Hughes, 2011). There are several possible benefits. Firstly, ipsative feedback closes the feedback loop in that learners can be informed on how far they have responded to previous feedback and progressed see fig 1. Such a scheme can initiate dialogue between learner and tutor without necessitating a meeting.

Fig. 1 Ipsative cumulative scheme for 3 sequential assessments

Feed forward 1 Feed forward 2

etc.

Feedback on feed forward 1 Feedback on feed forward 2 by comparing assessments by comparing assessments

1 and 2 2 and 3 Feed forward 2 can build on feed forward 1 if remedial action has not been taken.

Secondly, ipsative feedback is longitudinal and gives learners and their assessors a way of monitoring progress over time involving several assessments. This is particularly helpful in a modular system where learners can select modules which are not strongly interconnected. Finally, not all learners will be high achievers and ipsative feedback could be motivational for learners who do not receive immediate rewards for effort through high grades.

Assessment for distance learners and the role of technology

There are inevitably differences between assessment practice for on-site students and distance learners and provision of feedback is of particular significance. Differences between on-campus and distance assessment regimes are found in the administration of assessment, for example, submitting coursework remotely rather than in person or taking examinations in a distant location (Morgan & O’Reilly, 1999). Technological innovation has much to offer distance learners. Online courses which engage with a range of technologies may be more innovative in summative assessment methods, for example using e-portfolios or blogs, but this will probably vary with discipline. While written feedback is almost always provided at a distance, often electronically, verbal feedback arising from face to face tutorials and from teachers and peers in class may not have a direct distance learning or e-learning equivalent.

Ipsative assessment could help provide the missing dialogue for distance learners, but one of the main arguments against using ipsative feedback is that it requires much more organisation than conventional feedback because the assessor must have access to previous feedback and assignments to make comparisons. However, there are ways of making this easier using technology. For example, Nicol & Milligan (2006) suggest that if assignments, drafts and feedback are submitted electronically, then they could be stored in one easy to access place such as a virtual learning environment. Feedback comments could also be selected from electronic menus.

Assessment 3 Assessment 2 Assessment 1


4

Previous work on ipsative assessment for distance learners

A preliminary scoping study on ipsative feedback and distance learners (Hughes el al. 2010) aimed firstly, to establish the extent to which ipsative assessment is used on a distance learning Masters programme; and secondly, to explore both learner and tutor views on the benefits of using ipsative assessment. This study concluded that ipsative assessment was little used on this distance learning programme and tutors were not proactive in informing learners of progress. However, while learners were self-assessing on progress, they indicated that they would find ipsative feedback helpful and motivational. Tutors tended to provide generic feedback that was very criteria focused suggesting that obtaining a good grade is viewed as the primary purpose of assessment, but learners found much of this feedback vague, and specific examples of what to improve would be more useful for their learning. Hughes (2011) proposed that ipsative grading might shift the focus away from outcomes onto progress, but in the study ipsative grades were perceived to be problematic by tutors and many students. The study also recommended that there might need to be a built in mechanism for including ipsative feedback if the criteria referenced grading process remains unchanged.

Because ipsative assessment is not explicitly practised in higher education, there is little evidence for its potential benefits. Therefore this intervention project was established to introduce ipsative feedback into a distance learning programme and to evaluate the new approach. The intervention aimed to:

1. Discuss ipsative assessment and its potential benefits with the tutors and then agree with them a process for providing the distance learners with cumulative ipsative feedback.

2. Evaluate the impact of the ipsative feedback intervention process on motivation and learning from the perspective of both learners and tutors.

3. Consider the scalability, transferability and sustainability of the process.

Ipsative grades were not introduced as this would require changes to the institutions regulations.

3. Methodology and background to the project

This study builds on an investigation of a distance learning Masters programme: the MA in Applied Educational Leadership and Management (AELM), delivered wholly online to a fixed cohort at a research intensive university. This scoping study had focused on tutor feedback for the two core module assignments and included questionnaires to students, an analytical tool applied to feedback and an invitation to tutors to respond to the findings (Hughes et al., 2010). The second implementation phase of the study took place during the 2011 spring and summer terms, i.e. halfway through the programme’s cycle and focused on implementing an ipsative feedback process for the two core (year 1) and two optional (year 2) module assignments. A meeting of the research team, which included the programme leader, discussed the systems that might need to be in place to support the change in feedback. It was also agreed that it was important to involve the tutor team, at an early stage, and so the next programme academic group meeting included an item on ipsative feedback, where the purpose of ipsative feedback was discussed and the findings of the initial research were presented. As a result of the meetings and discussions, the programme leader instigated the development of a new Assignment Submission Form (Appendix 1) which would enable students to inform tutors of how they had acted on previous feedback and a new data base of current students’ assignment


5

mark sheets, located on the Virtual Learning Environment (VLE) accessible to tutors. Both of these interventions could provide tutors with a means of comparing current and previous assignments. Although developing the data base required extra work from the programme’s administrator, it was set up to be self-supporting with tutors uploading mark sheets. Data collection and analysis The sample The total sample of 28 mostly international students comprised 17 first-year students and 11 second-year students and 17 females and 11 males. The email interview questions were sent out to these 28 students and 13 replied. Out of 13, seven were first-year students and six were in the second year. The feedback forms and written summative feedback of 27 students were analysed (28 were selected but one deferred the assessment), including feedback items from six tutors. A list of students and the degree to which they participated is given in appendix 3. Three sets of data were collected: the use of the Assignment Submission Form and the written feedback received by students, email interviews with student volunteers and e-mail interviews with the tutors of these programmes. Draft questionnaires and interview schedules were shared amongst the research team before final versions were agreed. Analysis of completed forms and written feedback Draft and final module assignments were submitted to an ‘Assignment Drop-box’ in the VLE. Tutors retrieved students’ scripts, prepared their feedback and then upload it into the same space so that students could receive it. Assignments were double marked, but feedback was provided by the first marker who sometimes added additional comments from the second marker on a single form. An independent researcher used an analytical tool based on the Hattie & Timperley (2007) model for effective feedback, distinguishing between feedback on current performance and feed-forward for the next assignment. Criterion‐based and ipsative feedback

were identified and task specific and more generic comments were also distinguished (see appendix 2). The students taking only the option modules (most of second year students) were unable to retrieve the Submission Form from the VLE and therefore were not able to complete it. However, their feedback was analysed and some volunteered for interviews. Student Interviews Interviews were conducted by email by an independent researcher. Some students interviewed had not been able to complete the form but these results were informative and therefore were included in the study. Samples of ipsative and feed forward comments were sent to students to reflect upon as part of the email questions. The email questions (Appendix 4) were divided into three sections: 1) new assignment submission form; 2) styles of feedback, usefulness and motivation; 3) recognition of progress and ipsative feedback.


6

3. Tutor Interviews Five tutors were asked to respond to interview questions which were sent by the programme leader via e-mail correspondence after all 2010-11 assignments had been marked. The programme leader was judged to be more suitable than an independent research in this case because she had initiated the discussion and use of the form with the team. The programme leader, who tutored one of the optional modules and marked assignments across all optional modules, was also not included in the sample because she was part of the research team. Two tutored the core module, one tutored both core and optional modules, and two tutored optional modules. The two tutors, who tutored the optional modules only, were not able to find the time to respond to the questions so three of the tutors were interviewed (2 males and 1 female). The questions were developed out of the students’ responses to the specific feedback they had received. Four questions explored tutors’ approaches to the new Submission Form, their views on the use of the VLE to further students’ understanding of the assessment process, how ipsative feedback could be managed, and how the programme team might support tutors in developing an ipsative feedback approach. Tutors were also given the opportunity to make additional comments. The e-mail sent to tutors also included specific comments about use of peer feedback taken from the student interviews, for example, about the difficulty some students have interacting with others and their uncertainty about how to give peer feedback (See Appendix 5). Data Analysis: The three datasets were systematically organised for interpretation through thematising and summarising. The following themes emerging from the feedback analysis:

use of the form;

student response to previous feedback;

tutor response to actions listed on the form;

ipsative feedback without use of the form. These themes were then cross‐referenced with themes emerging from the student interviews:

the new Submission Form and usefulness;

motivation for distance learners;

short-term grade improvement;

continuity between tutors. Themes emerging from the analysis of tutor interviews were also cross-referenced:

change in practice for a new feedback approach;

supporting an ipsative feedback process;

supporting student understanding of assessment and managing expectations.


7

4. Results of analysis of the feedback form and written feedback

Numbers using Assignment Submission Form

11 students completed the ipsative feedback sections on the form entitled: ‘Feedback given on previous assignments’ and ‘Formative feedback on this assignment’. These students also provided an account of how they had addressed the previous feedback in the section entitled: ‘Response to formative feedback’. A further 5 students requested feedback on a specific issue, but did not complete the new sections of the form or in one case used an old form without the ipsative feedback sections. 11 of the 27 students submitting work did not complete the form. Non-completion was not obviously linked to achievement and from student accounts was mostly because there were technical problems downloading the form from the VLE for the option modules.

The ‘Request for specific feedback’ section had been introduced at the beginning of the year and so was more familiar to the students than the other ipsative sections. As some students filled in this section but not the new ipsative sections, it is reasonable to assume that more students might fill in the ipsative sections as they become more accustomed to the new form.

Student response to previous feedback

Those who filled in the sections about previous feedback tended to list points from both previous summative feedback and formative feedback for the current assignment. For example from student 1:

-to be more argumentative

-to draw more on relevant literature to inform and support the analysis, so it is less descriptive

-to ensure the assignment stays within the guidance provided on word length

In the ‘Response to formative feedback’ section some students gave a very general response to claim that they had addressed all the points raised. Student 11 claimed to have addressed all the points from previous feedback and wrote:

I have tried to act on all the feedback received.

This could indicate a superficial reflection on the part of the students and there is scope here for tutors to points out any over-inflated or inaccurate claims. In the example of student 1 above, the student claimed to have addressed all the points:

I feel I have attempted to address the points made, especially about the Literature Review and the Methodology to the best of my ability and understanding

But, the issue of word-length had not been addressed as this student incurred a penalty for exceeding the word-length. There was no ipsative feedback to highlight the inadequate response to feedback.

Others students gave details on how they had acted on the feedback. For example, student 22:

-In the academic review section I have added an additional section which focuses solely on external factors....

-In the methodology section I have added more detail and have given further thought to sampling and validity of data


8

Student 27 explained:

The ethical issues raised in the methodology section were resolved by changing the participation in the research from compulsory to fully voluntary

These students have taken note of the feedback and made confident changes to their writing and/or content but Student 13 was less certain and is clearly opening up dialogue:

I tried to show more connection of the literature to my research, but I’m not sure if it’s this the right way.

Tutor responses to student actions and provision of ipsative feedback

Three tutors (A, B and C) taught the core modules which used the new form. Their responses to the students’ claims to have acted on feedback were supportive and helpful:

- It is clear that you took careful note of the advice you received on the draft and the previous assignment. You have worked hard to improve the narrative flow and have taken steps to develop your critical engagement with the sources. (Tutor A).

This was definitely an improved version, compared to the draft you previously submitted for formative assessment. You have managed to shift the focus from a description of an intervention on enhancing school/family links (which was not the focus of this study) to a small scale study addressing one particular research question. (Tutor C)

In another case this tutor provided a positive ipsative comment but also added that there was more work to do:

Although this assignment was an improvement compared to the previous version you submitted for formative feedback and you have moved from discussing the establishment of a research centre to proposing a small scale research exploring staff and students’ views, there are issues that could be further addressed to improve your work further … establishing a strong narrative thread that will allow you to capitalise on your command of the field of study and second, ensuring that your referencing is appropriate (Tutor C)

The tutors had already agreed to provide ipsative feedback and the results suggests that the form was instrumental in prompting them by enabling students to open up dialogue with their tutors about acting on feedback and progress.

But, the results were not consistent: in 3 cases the tutor (tutor B) did not respond to the student claims about acting on feedback. More surprisingly in other cases, where students did not complete or were unable to access the form, there was still evidence of ipsative feedback.

Ipsative feedback without use of the form

The limitation of such a form is that tutors may be dependent on it to review previous feedback and thus rely on student interpretation of that feedback. A positive ipsative comment particularly requires comparison with previous work to see that an improvement has been made and this requires a new process for the assessor. But there was evidence that tutors provided feedback on progress, or lack of progress, even when the form was not used.

For student 8 there was a positive comment as well as continuing issues:

Occasionally I detect examples of overly long indented quotation which was also an issue in your first assignment. However, overall your writing has moved on in range and detail, both from the early outline draft and from your earlier assessed work. (Tutor B)


9

For lower achieving students such as student 20, ipsative feedback potentially provided motivation beyond a grade:

You managed to effect major improvement from the draft submission and have cleared the first hurdle. You have secured a pass grade and I recognise the huge commitment on your part in the face of adversity that has got you there. (Tutor A)

But for student 7 there was a warning:

It is a pity you did not have the chance to revise your assignment on the basis of the feedback provided earlier, as this would probably result in a better grade. (Tutor C)

If the students do not use the ipsative feedback form, then the tutor will need to refer back to previous feedback to provide an ipsative response. Comments such as the above provide some evidence that tutors had done so and tutor accounts confirmed this.

There was further evidence that tutors did not rely completely on the information provided by students on the form. For example, for student 27 the list of points to address from previous assessments was:

-to pay more attention to ethical issues

-to improve the conclusion to make it fit the requirements of the brief

-to include an appendix listing the questions for the questionnaires -cutting down the usage of the personal pronoun

But although tutor A recognised progress on the points above, the tutor also raised a new area of progress:

-It is good to see that you have moved on from the frequent, quite lengthy quotations of the draft, though perhaps you need to think carefully about how you develop a narrative thread which links together the ideas of various authors without it becoming a little like a list. (Tutor A)

The point about length of quotations does not come from the form, so either the tutor remembers this issue from the draft or has looked at the written feedback given for the draft and then added this comment.

In another example student 16 did not provide an account of previous feedback, but the tutor response was detailed:

this assignment was certainly an improvement compared to the previous version you submitted for formative feedback. You avoided repetitions (especially when referencing) in a revised Literature Review section, which appeared focused on your purpose. In addition, you provided a well-documented research design in support of your stated aims. (Tutor C)

Even with small numbers it would be unlikely that the tutor would remember this level of detail and again this tutor has probably referred back to the comments s/he wrote on the student’s draft assignment.

Referring back to one’s own previous feedback may not be unusual. However, in a fully ipsative scheme tutors might need to refer to comments written by other tutors. Tutors do not routinely make their feedback available to colleagues - giving feedback is possibly a more solitary activity than teaching - and there was no evidence of reviewing the feedback provided by other tutors in this study.


10

5. Results of analysis of student interviews

Relevant themes emerging from student interviews were: the new assignment submission form, feedback and motivation and short versus longer-term views of assessment. The Assignment Submission Form and its usefulness Students who used the form identified two particular aspects as beneficial. One was that the form allowed them to reflect on how and to what extent they had responded to tutors’ comments in the formative feedback. For example, one student commented: ‘It [the form] allowed me to show how I had attempted to improve on my last assignment and also encouraged me to reflect more deeply on my submission’ (Student 22). The other benefit was that the new form facilitated a systematic approach to keep track of changes made which enabled self-assessment:

The two step process was like a self assessment. The first step was how I had approached and understood the assignment. After incorporating my tutors’ inputs the second step allowed me to explain the changes I had incorporated. The difference between the two steps helped ‘focus’ the areas of change (Student 19).

There was one more sceptical comment about the new form being:

..for the convenience for tutors to check quickly what advice each student have received, and check what extent each student respond in their assignment’ (Student 14).

Student 14 continued to explain that the form would become useful only after the second time students fill it in – because the first time benefits only the tutor:

the new form is not useful for the students who use it for their first time…. I expect from the next time tutors will give me comments more in detail based on their deep understanding about my capacity to understand tutor’s comment and to respond it (Student 14).

But the student’s further comment suggested that there is a longer-term expectation of improved feedback through dialogue and this will be discussed further below. Overall, the usefulness of the new form was recognised by those who had used it, albeit a small sample. A tool such as this form has a potential to improve the mechanism of assessment by offering students an opportunity for reflection and to provide both students and tutors with a space for better communication. Feedback and motivation for distance learners Feedback was important for these distance learners, whether or not they received ipsative feedback. They often claimed to take it ‘very seriously and always act on it’ (Student 10). They were very dependent on their tutors for feedback as few mentioned peer feedback and those that did were uncertain of the value of peer feedback. Students intended to act on feedback but were sometimes unable to follow through on part-time distance learning course because of workload. In this case getting access to sources was also an obstacle:


11

I have been very pleased with the feedback received. It was clear, concise and helpful. I do not think it made reference to improvement with reference to the draft or previous assignment (maybe I did not improve) I tried to act upon the recommendations but at times the task seemed daunting when I could not get hold of additional sources to support my findings (Student 26).

The problem here is not about feedback, but about the learning environment. This student did not receive ipsative feedback and was easily daunted by lack of progress, but other students were very much aware of their progress and, as in the previous study, progress was linked to grades, but there was also awareness of improved learning:

I had specifically tried to improve by following the advice given and it was helpful to know my changes had my (made) a difference and reflected in a higher mark for my second assignment (Student 22).

There was evidence that ipsative feedback was particularly motivational by building confidence. 8 out of 13 students mentioned that they had received clear comments from tutors about how their work improved or not improved since the previous piece of work. With one exception, all received a comment saying the work had improved, which they felt ‘helpful and encouraging’ (Student 22). When tutors explicitly identify areas of improvement with reference to the previous version of the assignment, students seem to appreciate their own progress and to be able to build their confidence in acting on feedback in the longer-term:

Ipsative comment from tutor: this assignment was certainly an improvement compared to the previous version you submitted for formative feedback. You avoided repetitions (especially when referencing) in a revised Literature Review section, which appeared focused on your purpose. Tutee’s response: [the comment] gave me confidence and made me feel proud of my work – this was very important as it was the first assignment I had handed in and I was very unsure about it! It also made me aware of the changes that I made that improved my work so I will know what to do next time (Student 16).

As was the case in the findings from the previous research, generic guidance, such as ‘greater depth in analysis required’ (Student 13) was sometimes confusing. Specific comments and concrete examples were helpful: a ‘more direct type of feedback’ (Student 1) such as ‘you need to look at this’ or ‘if you are including this, then you should look at this’ (Student 24). Ipsative feedback was also appreciated alongside task relevant examples:

Ipsative comment from tutor: You are certainly writing with improving clarity but there some points at which a lack of easy fluency weakens the impact of your narrative. An example of this would be the paragraph on page 6 where good points based on Fidler needed better expression. Tutee’s response: Because improving clarity is one of the biggest issues for me, and [this comment] was helpful for me to understand this issue because tutor suggesting an example to be improved (Student 14).

But, although illustrations are clearly valuable, students’ preference for direct examples might stem from a focus on short-term grade improvement. Short-term grade improvement and long-term views on feedback


12

Many tutors’ generic comments addressed longer-term strategies but some tutees did not seem to appreciate such long-term views. Instead, several students were seeking to take action to quickly raise the quality of the essay:

My tutor [Tutor C] gave particularly good feedback after my draft assignment. It was easy to understand and follow and was very specific so I could see exactly what I needed to improve and how. I think her feedback made a big difference to my final mark (Student 16).

Many were disappointed not to gain extra marks by acting on feedback on draft work. One high achieving student wrote:

I do find that the generous positive comments, motivating as they are, don't always translate into higher marks. Perhaps the marking is very strict but after all the hard work, the positive feedback encourages hopes for higher marks and I wasn't quite sure why I eventually only grazed into an A grade. Grateful as I am to be doing relatively well, in this regard, given my exact same marking, I wasn't sure whether I had progressed at all during the year (Student 13).

There was further evidence of student expectations that acting on feedback would produce short-term rewards in terms of improved marks. In this example the student was also proactive in informing the tutor that s/he had acted on the feedback and did not agree with the ipsative feedback provided:

…the tutor reminded me of not having addressed an issue raised about my first draft. I did email the tutor and highlighted an extract from the final essay where I had covered the suggestion made. I believe the tutor had missed that point and the comment was then removed from my final feedback. I was told, however, that this would not result in a revised mark (Student 13).

However, another student was clearly under pressure to finish the essay, but was aware that the benefits might be longer term:

I wasn’t always entirely sure how to act on the feedback, apart from by trying to reflect on the advice/comments. To be honest, I felt more relieved to just to finish the essays in the first place! However, all the feedback has helped when I have gone on to do subsequent writing. I have tried to consider where I went off tangent before (Student 3).

This is useful as taking a short-term view of expecting higher grades immediately can be demotivating if expectations are not met. While students are motivated by grades, it would seem that introducing ipsative feedback does not necessarily engender a longer-term view of progress and learning, but this longer term view might develop as the students become more familiar with the ipsative process. Tutors were aware of this and had some suggestions.

Continuity between tutors

In response to the questions on ipsative feedback, a couple of students identified discrepancies between feedback on a draft and the summative feedback which came from a different tutor:

I received such [ipsative] comment once and it was from [Tutor C]. Other tutors would rarely follow up on feedback once given. In one incident, the final assignment’s results had little to do with the feedback given on the draft, and I lost some marks on points not raised in the initial feedback, which surprised me (Student 24).


13

Another recognised that tutors may not necessarily have access to previous feedback and this will prevent ipsative feedback:

As different tutors are responsible for different courses it may be difficult for them to do this [ipsative feedback]. Unless there could be a stock of comments kept on students previous essay work [by] previous tutors? (Student 3).

Receiving new feed forward in the summative feedback that had not been mentioned in the formative feedback was also criticised by some. For example:

After the first draft for both assignments, the first feedback from them was detailed. This helped me to reflect, redo and improve the assignment. The second feedback sometimes had additional findings which were not mentioned earlier (Student 19).

There is a dilemma here for tutors who fear too much feedback in one go might be counterproductive and take a ‘drip feed’ approach providing students with new areas for progress each time. Feed forward must be achievable and realistic. It is not helpful to provide lengthy feedback to a student who is a borderline pass on what they would need to do to achieve the top grade. However, it may be that the tutor missed an important piece of feedback first time round. Using the Submission Form might make such practice more visible and easier to change. Again this highlights a need to encourage learners to take a long-term view of developmental feedback and not focus on short-term marks which is difficult when the short-term marks are high stakes. Tutors were aware of this dilemma as we shall see in the next section.

6. Results of analysis of tutor interviews

The responses to the questions were categorised into three sections: change in practice to respond to the new form; supporting an ipsative feedback process and supporting students’ understanding of the assessment process.

Change in practice to respond to the new form

Each tutor responded differently to the new Submission Form. Although they agreed that it changed the way that they gave feedback, ‘up to a point’ (Tutor B), one tutor commented that:

I was already on the journey to that approach. When providing feedback I tended to approach it in my normal way and then made sure that I adapted things to meet the requirements of the new front sheet. (Tutor A).

Tutor C, who had been tutoring on the programme for a shorter period of time, had changed more radically by carefully checking whether each student had addressed the points made in previous formative feedback. This tutor also commented that ‘students had made significant efforts to address the points raised’ which is consistent with both the comments made by students, although less consistent with the evidence from the analysis of the written feedback which suggested that students may generalise claims to have addressed feedback or may only partially address feedback. An issue was raised about the process being introduced half way through the year. Student feedback had suggested that the benefits could be limited because they were half-way through a cycle, but tutors felt it gave them the opportunity to get used to a modified process and so they would be more confident in applying it at the beginning of the next year.


14

Supporting an ipsative feedback process Tutor interviews suggested that a positive outcome of the implementation of ipsative feedback was ‘further dialogue over the nature of progression’ (Tutor A) and in particular how ipsative feedback, once fully understood as a development tool, could contribute to students’ learning (Tutor B). This is consistent with some student reports of using the form for dialogue with tutors about their learning. A further benefit was the way in which information was shared amongst the tutor team. The core tutor team, who taught both modules, ‘invested a lot of time’ (Tutor A) in sharing information about individuals and their work, but there had been no mechanism in place to share this knowledge with tutors who took over for the optional modules. The tutors indicated a readiness to further explore the use of ipsative feedback, however, tutor B also warned, that it could increase the burden of work.

Providing systematic cumulative feedback is a desirable objective, but the system we adopt should be efficient as well as effective and not place unsustainable burdens on students of (or) tutors. (Tutor B)

This rather more sceptical view of use of the form, from Tutor B, is also consistent with the greater use of the form, made by Tutors A and C identified in the analysis of the tutor written feedback. Supporting students’ understanding of the assessment process

There was a belief that students’ expectations of the assessment process needed to be managed as these expectations could range from the ‘reasonable to the unreasonable’ with students immediately seeking solutions to raising their grades (all tutors).

I could see from the comments that some students expected their revised draft to lead to an A, and it was subsequently disappointing for these students to gain a sense of progress without an A. (Tutor C)

The tutors’ view that it was difficult to manage students’ expectations and deepen their understanding of the process was consistent with some student expectations of continuity between tutors and the dissatisfaction at not receiving an immediate grade increase when they acted on feedback. A further issue that emerged was the programme’s diverse intake (which the team views as a positive feature of the programme) and the impact this has on the student’s ability to understand that emerging issues are generic, irrespective of the context. This was an issue that would need careful consideration when developing and improving the ipsative feedback process. There was also concern about students’ understanding of the different requirements made by consecutive assignments and the different skills they would need to evidence. For example, the assignment for Core Module 1 focuses on developing the students’ ability to critique literature, identify themes and apply them to their own workplace, whilst the assignment for Core Module 2, additionally, develops the ability to design, but not carry out, small scale research. Tutor A commented:


15

The two [core module] assignments are very different in nature, draw on overlapping but not identical skill sets and each poses its own challenges. It is perfectly feasible to have picked up (in the student’s view) on earlier guidance only to fall down on something which is specific to the second of the assignments. Perhaps we need to make this clearer. (Tutor A)

Some students also pointed out that even when they addressed the feedback there was more to do even for the same assignment, and this supports a recommendation that assessment needs to be viewed as a longer-term developmental process where new challenges will arise. The tutors made suggestions for broadening views of assessment that included: putting together a statement of the goal/aims of ipsative feedback, exploring the addition of peer feedback to supplement tutor feedback, and revisiting the contribution of the VLE discussions to students’ understanding of the assessment process. There is evidence that reading student reflections on the form provoked these ideas but it is not possible to attribute all the development in tutors’ thinking about assessment to use of the ipsative feedback form becuase taking part in the research process may also have contributed.

7. Discussion of further implementation of ipsative feedback The findings above indicate that a formalised process for providing ipsative feedback has shifted the way in which some tutors write feedback and has been helpful and motivational for learners. This pilot implementation process is therefore worth scaling up and continuing. There are several points for further discussion. Use of a form to promote ipsative assessment Use of a form both to make the ipsative feedback process explicit and to involve students in a dialogue on feedback was effective. This is particularly important when ipsative feedback is new to both tutors and students. A dialogue between tutors and students is key to effective assessment (Nicol & Macfarlane-Dick, 2006) and for distance learners where there is little direct contact, use of a process such as this is very valuable. However, the form was not consistently used by students and some tutors did not provide ipsative feedback even when students did complete the form. Further investigation will determine whether this problem will disappear as both tutors and students become more accustomed to the ipsative process. Ipsative feedback did arise without the form being used and it may be that once a longitudinal approach to assessment becomes established, such a tool would be superfluous. Ipsative feedback as motivational but raising expectations There was evidence that ipsative feedback is motivational for students and there is some direct evidence of enhancement of learning through use of the form. However, the process used in this study also raised student expectations as student believed that if they acted on previous comments then they would gain immediate higher marks. A longer-term view of feedback will need to be established so that learners will view progress over time and not have expectations of immediate results. However, it might be difficult to persuade learners, who are accustomed to relying on marks, to gauge their progress and to view progress in the longer-term as the ultimate goal. It may be that introduction of some form of ipsative grading will be necessary (Hughes, 2011).


16

Making a comparison with previous assessments Comparison with previous assessment is problematic for two reasons. Firstly, access to previous feedback and assessment is not always easy and may be time-consuming for tutors. Inviting the students to provide the summary of previous assessments is helpful, but it may not be accurate and not all students may see the advantages of filling in a form which may appear bureaucratic. Other ways of comparing assessments are therefore worth exploring as well as use of technology to make the comparisons easier. Secondly, there is the problem of consecutive modules assessing different skills. This means that comparison is not straightforward. It can also lead to students’ disaffection because they believe that they have acted on previous feedback but the response they get from the assessor is that there is more work to do. Broadening understanding of assessment to view it as a longitudinal journey throughout a programme, where a learner is expected to make continual improvements is therefore imperative. Again, as above, this may be challenging in current competitive and summative assessment-led regimes, but the tutors’ suggestions for clarifying the ipsative process and using peer feedback may help. Scalability, sustainability and transferability

In the previous study of this programme (Hughes et al. 2010) there was very little evidence of ipsative feedback being provided. The introduction of the ipsative feedback form has significantly increased the amount of ipsative feedback given to learners, albeit from some tutors more than others. This is the first time the innovation has been tried and it might be expected that continued use of the form might increase the numbers of students engaging with issues of progress and encourage more consistent use of ipsative assessment by tutors through greater familiarity with the process and a greater understanding of the benefits. Tutors interviewed were willing to continue with ipsative feedback.

Numbers on this programme are relatively small, but using a form such as this means that there are possible scalable benefits in that keeping a record of feedback means that a different assessor could continue the process in future modules and students from much larger cohorts could be tracked. However, this would need a much more open and collegial approach to giving feedback. Further studies are needed to confirm the scalability of such an ipsative assessment process.

Using technology may provide some solutions to managing large volumes of student assessment data as Nicol & Milligan (2006) suggest. For example, making previous feedback and assignments readily accessible via a VLE or other technology might help students and tutors quickly identity progress. However, although a folder was added to the VLE containing this information, there was limited evidence that tutors referred back to previous assignments and feedback although some tutors reviewed their own previous comments. It may be that if ipsative assessment was fully embedded and became normal practice, then referring to previous assessments might be useful, especially when students do not provide information about previous feedback on the form or when student representations of their previous feedback and of their responses are inaccurate. Reference to previous assignments could, however, increase the time taken for the formative assessment process, but may be offset by encouraging students to act on cumulative feedback and thus needing less detailed feedback as they progress through the programme.

Although this study applies to Masters students, there is nothing to suggest that this approach might not also be successful for students at undergraduate level. Undergraduates are less likely


17

to be self-reliant than postgraduates and many take a similarly short-term and grade dominated view (Higgins et al. 2002). They may be less motivated to complete a form or less able to complete it accurately and tutors may need to be more proactive, but over the longer-time span of degree programmes the benefits of increased motivation and enhanced learning might outweigh this.

This was an education programme and other disciplines in social sciences and humanities could similarly benefit from an ipsative scheme such as this. But, that is not to say that scientific and technical disciplines might not also apply ipsative assessment to motivate students through making longer term progress visible, perhaps using a different methodology.

8. Conclusion and recommendations

This study has documented the second, implementation phase of a study of the potential for using ipsative feedback to motivate distance learners. A new assignment submission form was designed that enabled learners to reflect on addressing previous feedback and encouraged tutors to provide ipsative feedback, although use of the form was not essential. The ipsative feedback was well received by those students who completed the form and helped with both motivation and dialogue between tutor and students. However, a view of progress as short-term grade improvement limited the potential of ipsative feedback and although a few students appeared to take a longer term view of assessment, perhaps promoted by the new ipsative process, this needs more encouragement and embedding.

Further work will consider the following recommendations which emerge from this report:

1. Ipsative feedback potentially has motivational and learning benefits for both tutors and students and should be embedded on this programme.

2. The ipsative process at programme level needs to be made clear to students and inclusion of some ipsative grading and peer assessment might help.

3. The sustainability of ipsative assessment and longer-term approaches to assessment on the AELM programme should be monitored.

4. The use of technology to facilitate ipsative feedback needs to be embedded e.g. recording feedback in the VLE as a standard practice, but tutors also need to be encouraged to share feedback with colleagues.

5. Use of ipsative feedback could be tested with other programmes at different levels, scale and for different disciplines.


18

Bibliography

Bloxham, S. Boyd, P. & Orr, S. 2009. Mark my words: An analysis of HE lecturers’ essay marking approaches. Challenging Higher Education: knowledge, policy and practice. Society for Research in Higher Education conference papers, Newport, Wales.

Carless, D. 2006. Differing perceptions in the feedback process. Studies in Higher Education. 3, no. 2: 219-233.

Crisp, B. R. 2007 Is it worth the effort? How feedback influences students' subsequent submission of assessable work. Assessment and Evaluation in Higher Education 32, no.5: 571-582.

Falchikov, N. & Boud, D. 2007. Assessment and emotion: the impact of being assessed. In Rethinking Assessment for Higher Education: learning for the longer term, eds. David Boud and Nancy Falchikov, 144-152. Abingdon: Routledge.

Gibbs, G. (2006) Why Assessment is Changing. In Innovative Assessment in Higher Education, eds. C. Bryan and K. Clegg,11-22. Abingdon: Routledge.

Handley, K. Price, M. & Millar, J. 2008. Engaging Students with Assessment Feedback. FDTL5 Project Final Report, Oxford Brookes University https://mw.brookes.ac.uk/download/attachments/2851361/FDTL+Final+report+to+HEA+-+September+2008.pdf?version=1

Hattie, J. & Timperley, H. 2007. The Power of Feedback. Review of Educational Research 77 no. 1: 81-112.

Higgins, R., Hartley, P. & Skelton, A. 2002. The Conscientious Consumer: reconsidering the role of assessment feedback in student learning. Studies in Higher Education 27, no. 1: 53-64.

Hughes, G. 2011. Aiming for Personal Best: a Case for Introducing Ipsative Assessment in Higher Education Studies in Higher Education 36 (3): 353 - 367

Hughes, G., Okumoto, K. & Crawford, M. 2010. Ipsative Assessment and Motivation of Distance Learners. CDE Project Report http://cdelondon.wordpress.com/2010/07/28/use-of-ipsative-assessment-in-distance-learning/

Lizzio, A. & Wilson, K. 2008. Feedback on assessment: student’s perceptions of quality and effectiveness. Assessment & Evaluation in Higher Education 33 no.3:263-275. Morgan, C. & O’Reilly. M. 1999 Assessing Open and Distance Learners London: Kogan Page. Nicol, D. & Macfarlane,-Dick, D. 2006. Formative assessment and self-regulated learning: a model and seven principles of good feedback practice. Studies in Higher Education 31 no. 2: 199-218. Nicol, D. & Milligan, C. 2006. Rethinking technology-supported assessment practices in relation to the seven principles of good feedback practice. In Innovative Assessment in Higher Education, eds. Cordelia Bryan and Karen Clegg, 64-80. London: Routledge. Poulos, A. & Mahony, M. J. 2008 . Effectiveness of feedback: the students' perspective. Assessment and Evaluation in Higher Education 33 no. 2: 143-154

https://mw.brookes.ac.uk/download/attachments/2851361/FDTL+Final+report+to+HEA+-+September+2008.pdf?version=1

https://mw.brookes.ac.uk/download/attachments/2851361/FDTL+Final+report+to+HEA+-+September+2008.pdf?version=1

http://cdelondon.wordpress.com/2010/07/28/use-of-ipsative-assessment-in-distance-learning/

http://cdelondon.wordpress.com/2010/07/28/use-of-ipsative-assessment-in-distance-learning/


19

Sadler, R. D. 2005. Interpretations of criteria-based assessment and grading in higher education. Assessment & Evaluation in Higher Education, 30 no.2:175-194. Simpson, O. forthcoming. Technology Supported Assessment for Retention. In Lynn Clouder, Christine Broughan, Steve Jewell & Graham Steventon (Eds) Improving Student Engagement and Development through Assessment Theory and Practice in Higher Education. London: Routledge. Simpson, O. 2008. Motivating learners in open and distance learning: do we need a new theory of learner support? Open Learning 23(3): 159-170.

Walker, M. 2009. An investigation into written comments on assignments: do students find them useable? Assessment and Evaluation in Higher Education 34 no. 1: 67-78.


20

Appendix 1: New Assignment Submission Front Sheet (elements added for ipsative feedback are in italics. Spaces in text boxes have been removed)

MA Applied Educational Leadership and Management

Assignment submission form

*Please copy and paste this form into the first page of your assignment file*

Name

Student ID

Module title

Title of assignment

Assignment word count (excluding references and appendices)

Is this a draft?

Yes / No (delete as appropriate)

Is this a resubmission?

Yes / No (delete as appropriate)

Date of submission

If this is later than deadline, please give submission date agreed with course director

Please indicate what feedback you were given, for your last assignment, in terms of how you could improve:


21

Please indicate what feedback you were given, for your draft of this assignment, in terms of how you could improve (if applicable):

Please comment on the extent to which you feel you have responded to feedback:

If you would like feedback on any particular aspects of your current assignment please make a note of them here:

CHECKLIST: Before you submit your assignment please check that you have followed all the guidelines so that we can identify, print and mark your assignment. Mark each box with an X symbol.

Have you:

saved your document with your name and module in the title (e.g.”SJacksonMod1assign”)?

inserted page numbers ?

remained within the word length limits stipulated for the assignment ?

included your word count on the submission form ?

set left + right margins to at least 2.4 cm?

followed guidelines for referencing ?

read and understood plagiarism code ( p58 - 61 of handbook)?

completed the feedback section of the submission form

completed an online course evaluation (for final assignment not draft)?


22

Appendix 2: Analytical Tool

Module:

F

E

E

D

B

A

C

K

S

H

E

E

T

Feedback given on previous assignments

Formative feedback given on this assignment

Response to formative feedback

To all or some

How

Other

Request for particular feedback

Student’s request

Tutor’s response

Comparison with assessment criteria or standards

Comparison with past performance e.g. comment on student's response to feedback on Feedback Sheet

SUMMATIVE

FEEDBACK

Grade:

1st-marker:

Task level

Generic level

Task level

Generic level


23

Appendix 3: Details of students involved in the study

Student number Completed form Y/N Interviewed Y/N

1 Y Y

2 Y N

3 Y (request for feedback section only) Y

4 N N

5 Y Y

6 Y (old) N

7 N N

8 N N

9 N N

10 N Y

11 Y Y 12 Y (old) N 13 Y Y 14 Y Y 15 N Y 16 Y Y 17 N N 18 N Y 19 Y Y 20 N N 21 Y N 22 Y Y 23 Y (request for feedback section only) N 24 N Y 25 N N 26 N Y 27 Y N 28 deferred

Totals submitted assignment(s) 27, completed form 15, no form submitted, 12, interviewed 13 (7 used form and interviewed, 6 did not complete form and were interviewed).

The breakdown was as follows: Core modules: no. of students Leading and Managing Educational Change and Improvement 6 Leadership for the Learning Community 7 Optional modules: Developing Leadership and Management Skills 5 Exploring Educational Policy 2 Values, Vision and Moral Purpose in Educational Leadership 3


24

Appendix 4: Student Interview Questions

Ipsative Assessment Project Email Questions These questions are part of a research project investigating new assessment arrangements on your programme. The changes have been made in response to tutor and student interviews from last year. The questions mainly refer to the modules listed below. Please give as much detail as you can. We might wish to follow up the answers you give. Your answers will be confidential and anonymous and will not in any way influence the marking of your work. Please tick the assignments you have submitted during this academic year. Leading and Managing Educational Change and Improvement Leadership for the Learning Community Developing Leadership and Management Skills Exploring Educational Policy Values, Vision and Moral Purpose in Educational Leadership New Assignment Submission Form

1. Have you used the new Assignment Submission Form (cover sheet)? If NO, please go to 4. 2. If you have found the new form useful, please explain in which way. 3. If you have not found the new form particularly useful, please explain why. 4. Please give any reason/s why you did not use the new form.

Styles of feedback, usefulness and motivation

5. Who gives you useful feedback, e.g. your tutor, other students, or do you prefer to give yourself feedback? Please explain your choice/s.

6. What is the most useful feedback you have received on the modules? 7. Here are two pieces of feedback you were given. Was either of comments more helpful than

the other? If so, why? Did one of them motivate/encourage you more than the other? a) An example of ipsative feedback. b) An example of feed forward feedback. How did you, or are you intending to act on the above feedback comment?

Recognition of progress and ipsative feedback

8. Did you receive clear comments from your tutor about how your work has improved or not improved since the previous piece of work? If so, was it helpful to receive such comments?

9. Did tutors inform you when you have not addressed the issues raised in previous feedback? If so, was it helpful to receive such comments?

10. What informed you of progress during the modules?


25

Appendix 5: Tutor Interview Questions and letter

Dear Colleagues,

If you remember, we have been trialling ipsative feedback to assignments submitted in the spring term, with a view to setting up a system that is simple for both tutors and students to use, and that would hopefully benefit all.

We have received the responses to the questionnaires and interviews from the students. They have proved very interesting, and the insights they are providing should inform development of the programme in several areas, not just assessment. I have attached the interview comments in a separate document.

The issues that are emerging are:

Draft and final assignment feedback

Feedback should comment on the student’s ability to explore concepts as well as their development of writing skills.

Comprehensive feedback for the first assignment, at draft and final assignment stage, is very helpful in understanding how to improve next time.

The use of examples is helpful in understanding good practice and how improvements can be made.

A mix of comments, about what is good work and where to improve next time, is motivating.

There is not always consistency between feedback for the draft and for the final assignment, i.e. new issues emerge. This is both confusing and demotivating.

It is demotivating to act on guidance and see no change in grade.

VLE activity

There appears to be variability in the response to how useful on-line activities are, with students finding it difficult to interact with/help each other (see the responses at the top of page 3, in section 5: who gives you useful feedback?)

Further issues that we are aware of and trying to address are:

The need for more specific research methods training.

The problems of writing an assignment when starting a new module.

I would find it very helpful if you would read the attached summary of comments about ipsative feedback and then answer the following questions. Your responses will inform the research and help me to think about how I can support your work as tutors and develop the programme to meet students’ needs. Your responses will be treated as confidential and if we want to quote you, we will first ask your permission and ensure the quote cannot be attributed to you.


26

These are the questions that we would like you to consider and respond to:

1. Did you find that the new front sheet changed the way that you gave feedback? Please explain your response.

2. How have the comments from section 5 (see highlighted comments below), informed your thinking about the contribution of the VLE to students’ understanding of the assessment process?

3. How would you suggest that we emphasise that ipsative feedback is for long term and not necessarily immediate benefits?

4. Is there any way in which the programme team might support you in developing an ipsative feedback approach to the assessment process?

5. Do you have any further comments that you would like to make?

Student feedback was inserted here

Thank you very much for your help, we will share our report with you when it is complete.

6. How would you suggest that we emphasise that ipsative feedback is for long term and not necessarily immediate benefits?

7. Is there any way in which the programme team might support you in developing an ipsative feedback approach to the assessment process?

8. Do you have any further comments that you would like to make?

Student feedback was inserted here

Thank you very much for your help, we will share our report with you when it is complete.

implementing ipsative assessment - university of london ... · pdf fileimplementing ipsative...

Documents