Assessing Students & Giving Feedback

Sean Polreis

Educational Support & Development

Council of Health Science Deans Office, University of Saskatchewan

Objectives Distinguish between formative and

summative assessment

Discriminate between various assessment instruments

Decide on appropriate ways to assess students.

Objectives

Learning ActivityGoal is to improve performance

FormativeFeedbackGames & Puzzles

SummativeFinal markGradePass/Fail

Assessment

CanMEDS Assessment Tools

Handout

1. Multi-Source Assessment

Who should give it?

What are the benefits?

Why are residents so negative about peer and patient (and other) assessment?

How can we overcome this fear?

2. Direct Observation

The learner is observed performing

work which occurs naturally in real

clinical settings during their training

period.

Direct Observation Advantages Authentic assessment Assess wide variety of skills – nearly

all key competencies of CanMEDS roles

Variety of perspectives/assessors is possible

Assess higher order behaviours – may be difficult to assess in exams

Very useful as formative assessment – immediate feedback usually possible

Direct Observation Disadvantages Standardization difficult (validity

suffers)

Multiple perspectives – different standards of performance

Behaviour varies with little control over the situations observed

Observation Instruments

Field Notes

Checklists

In-Training Evaluation Reports

Selectivity□ Focused/appropriate□ Establish priorities□ Urgent vs. non-urgent□ Complete, thorough Clinical Reasoning

Clinical Reasoning □ Hypotheses / Diff. Dx□ Gather data (Hx & Px)□ Interpret data□ Make Decisions□ Set goals / Objectives

Professionalism □ Responsible/Reliable/Trustworthy□ Know Limits□ Flexible/Resourceful□ Evokes Confidence□ Caring/Compassionate□ Maintains Boundaries□ Respectful□ Ethical/Honest□ Evidence Influenced□ Community Responsive□ Good Balance□ Mindful Approach

History Physical Hypothesis Investigation Diagnosis

Procedure Management/Treatment Referral Follow-up

Date: __________________ Resident ______________________ Supervisor ________________________

Patient: M F AGE______ Diagnosis/Procedure ____________________ Directly Observed □Yes □ NO

Patient centered approach□ Explores illness /disease□ Understand context□ Common ground□ Build relationship□ Be realistic□ Health promotion / prevention

Procedural skills□ Decision to Act □ Informed Consent□ Preparation □ During procedure (comfort/safety)□ If problems reevaluate □ After Care/Follow-up

Physical Exam□ Appropriate Exam / Technique□ Competent Performance

Communication□ Verbal□ Written & Charting□ Listening□ Non-Verbal□ Culture/Gender/Age Appropriate□ AttitudinalBased on Dalhousie University’s work

Field Notes

Continue (Strengths):

Do More (Area for Development):

Consider (Learning Objectives):

Stop (or do less):

Reporter Interpreter Manager Educator Procedure competence achieved

Resident’s Initials ____________ Supervisor’s Initials ______

Field Notes (Back)

3. Rubrics

Rubrics are explanations of

assessment which reveal the “scoring

rules” – the criteria against which work

will be judged. If shared with the

learner it gives guidance learners can

use to develop, revise, and judge their

own work.

Advantages of Rubrics Gives learners information on how

to improve – formative Describes what constitutes good or

excellent performance – real life standards of the profession

Reinforce intended learning outcomes

Guides instructor feedback to be specific

Summative – reliability enhanced

Questions to Consider What would constitute high quality

work?

What components contribute to quality?

What is substantial vs. superficial?

What are the consequences of poor performance?

Other Considerations Use objective descriptions to help

give guidance

Use words that describe qualitative differences – not comparative words (like “more” or “less” of a certain quality)

Example: Exemplary – Proficient – Acceptable – Unacceptable

Rubric Steps

1. Essential criteria required for high quality workI. General abilities and skills

II. Specific knowledge/content

2. Levels of achievement – number and descriptor

3. Performance consequences at each level

4. Rating scheme – scale and weighting

4. Portfolios

A portfolio is collection of materials and documentation which provides evidence of learning - skills and knowledge. It offers proof that competencies have been attained.

Supports Learning

Demonstrates

Competence

What’s in a portfolio? Case histories

Questionnaires used

Images and video

Projects

Teaching sessions attended

Presentations

References/testimonials

Types of Portfolios

1. UnstructuredOnly learning goals are defined

2. StructuredType of activity needed to achieve the

competency and documentation required are also predetermined

3. Semi-StructuredBalanced approach combining #1 & #2

(likely best)Provides some freedom

Portfolio Purpose

1. Formative- Reflection- Identify areas requiring work

2. Summative

- Certifying predetermined

competencies

3. Mixed

- Useful for both instructor and

learner.

Steps Involved

1. Define competencies – list required skills and knowledge

2. Create timeline

3. Describe actions required to acquire competencies

4. Assessment criteria - Expectations and standards

- % for various areas

Portfolio Advantages

Authentic – real activities of residents Incremental/Growth – progress over

time Learner-centered – multiple ways to

demonstrate achievement Focus and direction provided at outset Encourage learner responsibility Encourage reflection Broad and comprehensive – beyond

medical expert role

Portfolio Disadvantages Complexity – content and

competencies must be formulated at outset

Time consuming – planning, compiling, and assessing

Content overload – selectivity important – only include materials that facilitate assessment and demonstrate competencies

Difficulty converting qualitative information into quantitative data

Assessment Summary It is important to choose an

assessment method that is appropriate for the role/competency or objective

Multi-source, direct observation, rubrics, and portfolios have unique ways of assessing a variety of skills and knowledge

These methods can be used in both formative and summative assessment

Giving Effective Feedback

Objectives:

Describe the benefits and importance of effective feedback

Using the mnemonic THANCS, describe the important factors involved in giving positive feedback

Describe how feedback changes based on the learner’s location in the learning cycle.

Goal of Feedback?

The goal of feedback is to improve learner performance.

Receptivity of the learner is important.

Supportive learning environment

Trusting relationship

Separate behaviour from the person

Use descriptive, non-judgmental language

Barriers? Complex skill

Giving negative feedback

Insufficient time

Insufficient data

Using secondary data Less time available to observe learner

Types of Feedback

1. Minimal – Emotional

2. Behavioural – One Way

3. Interactive – Goal Based

Feedback (THANCS)

T imely

H elpful

A ppropriate

N ever labelling or demoralizing

C ollaborative and culturally sensitive

S pecific

Feedback Videos

Discuss effective and

ineffective feedback methods

based on these videos.

http://www.practicalprof.ab.ca/observation_feedback/giving_feedback.html

Theoretical UnderpinningDepending on how the student evaluates themselves, they are …

UnconsciouslyIncompetent

Video observation360 FeedbackStopStart

Consciously

Incompetent

Identify BehaviourStartDo moreStop

Consciously

Competent

EncourageContinueDo moreTeach others

Unconsciously

Competent

Feedback Videos

Identify strengths and weaknesses

in the following videos.

http://medicine.usask.ca/tips/giving_feedback.php

Feedback Steps

1. Mutually set goals and expectations – prepare for feedback at the start of a rotation

2. Select time to meet

3. Have learner self assess – listen carefully – probe and respond

4. Collaborate on solutions

5. Assess understanding/acceptance

Feedback Summary Since the goal of feedback is to

improve learner performance, what factors are important:

Collaborate - interactiveBe specific – helpfulGive in a timely mannerSupportive environment – not personal Identify learner weaknesses and

strengthsLearning cycle awareness