formal appraisal of undergraduates – worth the effort? deborah murdoch-eaton professor of medical...
TRANSCRIPT
Formal Appraisal of Undergraduates –worth the effort?
Deborah Murdoch-EatonProfessor of Medical Education
Leeds
http://www.ltsn-01.ac.uk
The workshop
• Who are we?
• What do you want to get out of the day?
http://www.ltsn-01.ac.uk
Proposed Outcomes
• Explored whether formal appraisal might be of benefit for students &/or faculty
• developed a model for undergraduates
• Considered whether it might be applicable in your institution ie “worth the effort”?
http://www.ltsn-01.ac.uk
What is appraisal?
• 1 to 1 interview
• Purpose?
(Sam Scallan review of educational appraisal)
http://www.ltsn-01.ac.uk
Purpose of Appraisal
• Review performance
• Provide opportunity to discuss progress
• Develop personal and professional development needs
• (contribute to University requirements for PDP)
http://www.ltsn-01.ac.uk
Is appraisal different for UG and PG and CPD?
• Purpose ?
• Type of goals set?
• Skills of reflection ?• Students?• Appraiser
• Appraiser skills
http://www.ltsn-01.ac.uk
Group work
• Benefits to student
• Benefits to faculty
• What stage of UG career might it be considered most valuable?
• Who should do the appraisals?
http://www.ltsn-01.ac.uk
• Feedback from groups
http://www.ltsn-01.ac.uk
Benefits to student?
• Individuality• Reassurance re progress and grades• Life at university• Feedback – on more than academic
criteria
• Achieve potential• Model for future ? Reflective practice?
http://www.ltsn-01.ac.uk
“A more positive feeling about how I am doing. I truly believed I was doing badly, and without this appraisal I would have maintained this disheartening and negative feeling”
http://www.ltsn-01.ac.uk
Benefits to faculty?
• Detection
• Alternative view on course
• Enhance student support system?
http://www.ltsn-01.ac.uk
Who gets appraised
• Does it depend on the medical school
• Type of curriculum?
• All students?
• Selected?• Performance• Year• Self-need
http://www.ltsn-01.ac.uk
Who are the appraisers?
• Different from PG and / or CPD model?
• Chosen• Knowledge of curriculum• Interest / understanding
• trained
http://www.ltsn-01.ac.uk
Coffee break
http://www.ltsn-01.ac.uk
methodology
• When do you do it?
• Preparation
• Documentation of outcomes?
http://www.ltsn-01.ac.uk
methodology
• How and when
• Preparatory material• Student• Appraiser
• outcomes
http://www.ltsn-01.ac.uk
Leeds Model
• When• After Easter break
• Who• All 1st and 2nd years, “selected” 3rd years• Is it “optional”?
(Murdoch Eaton & Levene Med Teacher 2004)
http://www.ltsn-01.ac.uk
Student preparation
• Written guidelines
• Preparatory questions
• Summative and formative assessment marks• Individual
• (Study habits survey – data from 2002 + 2003
available)
http://www.ltsn-01.ac.uk
Preparatory questions for students – work / course
• do you think the results you obtained were a fair reflection of the work you put in?
• Think about how you worked – was it always appropriate and efficient? Give an example of where it went well, what you are pleased about and why.
http://www.ltsn-01.ac.uk
Preparatory questions for students work / course
• Is there anything over the year you didn’t do as well in or learn as much from as you could have? How might you approach something like this differently again?
http://www.ltsn-01.ac.uk
Preparatory questions for students – career choice
• Has anything this year changed your views on what being a doctor is about?
• Either made you less or more enthused about your career choice?
http://www.ltsn-01.ac.uk
Preparatory questions for students – personal change
• How would you measure a successful next year? What would you like to do to ensure this happens (both personally and in your UG career)?
• Is there anything else you would like to discuss – personal or professional development?
http://www.ltsn-01.ac.uk
Study Habits Survey
• Who knows how hard a student works?
• What is the “norm”?
http://www.ltsn-01.ac.uk
Study Habits Survey
• One representative week in term 2• Document
• Number of hours private study(all course related activities which are unsupervised)
• Where, with whom
• Employment?• Hours/week on sport / social activities?
(Whittle S R & Murdoch Eaton D G. Study Habits audit. Medical Education 2004 (38) 566-567)
http://www.ltsn-01.ac.uk
Study Habits SurveyNumber of Hours spent on self-directed learning
0
5
10
15
20
25
30
0-5 6-10 11-15 16-20 21-25 26-30 31-35 36-40 >40
Hours per week
Nu
mb
er
of
stu
den
ts
Series1
http://www.ltsn-01.ac.uk
Study Habits SurveyHours spent sports / societies
0
5
10
15
20
25
30
0 1-2 3-4 5-6 7-8 9-10 11-12 >12
Hours per week
Nu
mb
er
of
stu
den
ts
http://www.ltsn-01.ac.uk
Use of Study Habits Survey Results
• Not the absolute values for comparison
• Relative in relation to• Academic results• Outcomes• Personal objectives
• i.e. A diagnostic tool??
http://www.ltsn-01.ac.uk
Personal Tutor involvement
• Sent copy of students results
• Form to complete• Relevant information• Not breaking confidentiality
http://www.ltsn-01.ac.uk
For the Appraisers
• Environment• Information for appraisers
• Student specific: Individual results appraisal record – (yr 2
+ 3)
PT report• General : preparatory guidelines prev results study habits survey
• Appraisal record form to complete with student
http://www.ltsn-01.ac.uk
methodology
• How and when
• Preparatory material• Student• Appraiser
• Recording the outcomes
http://www.ltsn-01.ac.uk
APPRAISAL RECORD FORM – year 1
STUDENT NAME APPRAISER DATE TOPIC OUTCOME NOTES Overview of the year’s academic progress, including assessment outcomes
Positive evaluation: What went well during the year, what achieved of last years’ objectives
Personal learning needs identified
Activities planned / to be considered over next academic year
Student signature Appraiser signature Refer to Dean
http://www.ltsn-01.ac.uk
Qns on bottom of Appraisal record form
• Option to refer to Dean / Sub-Dean of students
• Use of progress file
• Contact / met with Personal Tutor
• Consider for 3rd year appraisal
http://www.ltsn-01.ac.uk
Appraisal record form
What happens to the information?
• Student
• Personal tutor
• Student file
• Signed by both appraiser and student
http://www.ltsn-01.ac.uk
Lunch break
http://www.ltsn-01.ac.uk
Issues raised in appraisal
2 main outcome themes
• altered perception on personal progress
• Altered perception of faculty
…….feel like a person, not just a statistic – the Med school
does care!
http://www.ltsn-01.ac.uk
Video clips
http://www.ltsn-01.ac.uk
Tea break
http://www.ltsn-01.ac.uk
So is appraising undergraduates worth the
effort?Cost effectiveness ?
Students ~ 600Appraisers ~ 28
84 X 3 hour sessions = 252 hours senior academics time
Administration - complex
http://www.ltsn-01.ac.uk
Appraisers perspective
• all would do it again
• Personally rewarding
• Timing about right
• Training of future appraisers?
http://www.ltsn-01.ac.uk
Effect on Students?
Student Feedback
• Questionnaire• Quantitative and qualitative questions
Student Outcomes
• Analysis of Record of Appraisal Outcomes
http://www.ltsn-01.ac.uk
Outcomes – student perspective
Questionnaire – quantitative analysis
response rate (86%)
• 96% well informed about process
• 99% they could be honest in answers• 99% topics covered were relevant• 96% feedback was useful and accurate
http://www.ltsn-01.ac.uk
Outcomes – student perspectiveQuestionnaire – quantitative analysis
(2)
• 94% able to discuss academic problems• 98% appraisal record fair representation
of the points discussed
• 62% appraisal feedback would alter their study habits
• 33% appraisal feedback would alter attitude towards medical career
http://www.ltsn-01.ac.uk
Outcomes – student perspectiveQuestionnaire – qualitative
analysis
• Majority very satisfied with appraisal session
• Perspective of medical school changed• “feel like I’m a person, not just a statistic”• “the Med School does care”
http://www.ltsn-01.ac.uk
“2 things you will take away”
• Personal• Confidence, reassurance
• Work/social balance• Healthy balance
• Study skills advice• Time management, efficiency of working
• Academic progress• C grade is fine
http://www.ltsn-01.ac.uk
“Two things you will take away”
“A more positive feeling about how I am doing. I truly believed I was doing badly, and without this appraisal I would have maintained this disheartening and negative feeling”
http://www.ltsn-01.ac.uk
But does it change anything?
Longer term study
To identify from Appraisal Record Forms• Pattern of advice and areas discussed
• Year 1 appraisal• Year 2 appraisal
• What of year 1 goals are achieved by year 2• What are new goals for year 2, and what
goals are carried forward from year 1 to 2• Correlation between factors
http://www.ltsn-01.ac.uk
Analysis of Appraisal Outcomes
2 cohorts of students completed both 1st and 2nd year appraisals N=511
1. Anonymous student feedback questionnaires(88 % response rate)
2. Appraisal record forms (100% completion)
http://www.ltsn-01.ac.uk
Feedback questionnaire - key points
Overwhelming positive perception of the value of appraisal in both years
• 59.2% overall considered issues covered would alter study habits
• 51% of 2nd years considered they had made changes based on previous years appraisal
http://www.ltsn-01.ac.uk
Practice Points – key advice required early in undergraduates’ career
• Study skills guidance was the commonest need
• Some significant correlations were seen between types of advice required
• Evolution of personal goals - changing course requirements and maturation
• Early opportunity to address healthy work-leisure balance
http://www.ltsn-01.ac.uk
Undergraduate appraisal
• Worth the effort?
• We think so
• Data would suggest that it is, from both students and faculty perspective
• The future?• Later years????
http://www.ltsn-01.ac.uk
• Any questions / comments
Thank you for coming and participating! I hope you found it valuable.