lee atkinson-mcevoy, md glenn rosenbluth, md ucsf ... · lee atkinson-mcevoy, md. glenn rosenbluth,...
TRANSCRIPT
Lee Atkinson-McEvoy, MDGlenn Rosenbluth, MDUCSF Pediatric Residency Training Program
Planning to implement in your program? Learners (students , residents or fellows)
have used it at other institutions and are asking for your participation?
Future of medical education, want to start within your program?
Introduction Portfolio vocabulary Examples of IDPs in electronic portfolios How to create a IDP in an electronic
portfolio platform Are they worth the effort?
Introduction What is the purpose of this workshop? Why are we talking about portfolios?
(Why is this culture emerging?)
Understand the purpose of portfolios and how they can be used to create Independent Development Plans for learners.
Introduction to the process of setting up a portfolio for learners
Create a work book to use to implement an IDP within your program.
Introduction Portfolio vocabulary
“A portfolio is a purposeful and longitudinal collection of tangible evidence of learner-selected work that exhibits the learner’s efforts, progress or achievement.
This portfolio features the criteria for selection and judging merit, and includes evidence of learner reflection.”
UCSF EPIC Report, Adapted from the Committee on Student Assessment’s portfolio definition from Reckase, 1995.
Formative or Learning Portfolio: Portfolio used to aid learning, with accompanying feedback to improve performance
Summative or Assessment Portfolio:Portfolio used for evaluating and / or grading a learner’s performance for the purpose of determining their readiness for advancing to the next phase of an educational program
Work products that learners can use to document competency
Should be embedded within their work. Should be part of curriculum Examples: Test results Poster/abstract/journal article Letter from a patient or a patient's family member Resume/CV Select narrative assessments from E*Value Slides from a lecture Blog entry or essay about experience with patient
Critical reflection: is analysis of personal experience to enhance learning and improve future behavior and outcomes. Without it, portfolio would not be a portfolio but
a database. By critically reflecting on the artifacts in their
portfolios, learners identify their own strengths, opportunities for improvement, and development over time.
Goes beyond “consideration or meditation” A skill developed over time for learning and
lifelong professional development. Not just what happened (anecdote) More than your personal opinion of events Requires Data gathering Integration of past, present and future
experience Analysis and reframing
Adapted from Louise Aronson’s presentation on Critical Reflection
Critical reflection should describe how a piece of evidence is related to the
educational competency the learner is addressing what was learned about the competency what the learner discovered about themselves
(as a learner, clinician or human being) from the experience how this discovery can be used to further
professional development
Presentation Portfolio (or view): Display of evidence chosen by learner with reflection, organized and communicated to an advisor, mentor or others for a specific purpose
Learning Plan: is a learner-specific program or strategy for
learning Integration of critical reflections based upon
evidence for all areas into a plan which informs an efficient process for learning
Review of the learning plan by an advisor, mentor or program/course director.
Should have specific criteria, guidelines by which to assess where the learner is within the program.
If part of a summative evaluation some formal rubric should be used.
This information should be available to the learner to assist in the critical reflection and development of a learning plan.
Introduction Portfolio vocabulary Examples of Individual Development Plans
in electronic portfolios
Introduction Portfolio vocabulary Examples of Individual Development Plans
in electronic portfolios How to create a Individual Development
Plan in an electronic portfolio platform
Portfolio Step* Responsible party
1. Collect Evidence Program/trainee
2. Reflection Trainee
3. Evaluate evidence Mentor/Trainee
4. Defense Trainee to program
5. Summative Decision Program
* Friedman Ben David, M, Davis, MH, Harden, RM, et al. (2001). AMEE Medical Education Guide No. 24: Portfolios as a method of student assessment. Medical teacher, 23(6), 535-551.
What are residents to document in their portfolio?
Why document this way versus another way?
How will it be used? Formative vs. summative feedback?
Who will have access to it?
ACGME Competencies Program Specific Professional
Competencies Career Development Personal Development Learning Plans
Learners need a guide for what is expected with each entry.
How to pick evidence—criteria for selection What supporting information can be used
as evidence.
Provides context for artifact. Guides mentor/advisor in giving feedback Allows for evaluation on summative
feedback for assessment of competency.
Necessary for standardization of assessments.
Helps guide the learner in identifying where they should be for their level of professional development.
Rubric useful tool to standardize assessment
Rubrics: multidimensional scoring guidelines used to provide consistency in evaluating learner work elucidate scoring criteria so multiple evaluators,
using the same rubric, would arrive at same score or grade
provide a measurement system for specific tasks and are tailored to each project
let learners know what is expected of them provide learners the opportunity to do self-
assessment to reflect on the learning process
Does not meet Needs improvement Meets Exceeds
Selects evidence that demonstrates current progress relative to the milestone
Evidence does not provide information on the skills/ activities listed in the milestone
Appropriate evidence selected, but provides limited details to be able to infer progress toward the milestone; no information in reflection that provides context for the evidence
Evidence demonstrates the learner can perform at the level of the milestone or has a plan that makes achieving the milestone likely
Evidence supports achievement of the milestone and has characteristics of year two milestone
Provides a thoughtful reflection and assessment of the learner’s progress toward competency in the domain
Describes a situation that does not address domain skills; perhaps focusing on someone else’s situation; does not take the reflection seriously
Discusses a specific experience, but does not indicate critical reflection; no lessons learned are drawn from the evidence/experience
Learner described the value of the experience/evidence and articulated a fundamental lesson learned
Learner described the experience and addressed multiple and/or complex lessons learned.
Describes how feedback has informed learning in the domain.
Does not indicate what feedback was obtained to help develop competency toward milestone.
Description relies solely on personal perception about what and/or how to improve in this domain.
Based on own perceptions and one other source formulates a plan that is consistent with evidence provided for this domain.
Indicates more than one source of feedback and clearly describes how that information has guided learning.
Describes strategies for future growth in the domain
Not clear what the learner’s strategies are for future growth
Describes strategies for future growth but does not articulate specific resources or plans
Clearly identifies relevant next steps for growth in the domain including resources, strategies that are timely and feasible
Describes specific, timely, and feasible plan for future growth in the domain and considers personal and professional challenges, and strengths
Benchmarks: A reference point that serves as a basis for judging whether a student has met the associated educational standard.
Hardware—computers Software—electronic portfolio platform Time—orientation for residents and faculty,
time to complete portfolio, time to review/evaluate entries
Manpower—faculty to review, administrative staff to collect data, reminders etc.
Faculty Should know what is expected of the learner Give feedback in a timely fashion Use a rubric if provided to guide feedback
Peer(s) Can be used to provide feedback Must follow the same guidelines as the faculty
Portfolio Step* Responsible party
1. Collect Evidence Program/trainee
2. Reflection Trainee
3. Evaluate evidence Mentor/Trainee
4. Defense Trainee to program
5. Summative Decision Program
* Friedman Ben David, M, Davis, MH, Harden, RM, et al. (2001). AMEE Medical Education Guide No. 24: Portfolios as a method of student assessment. Medical teacher, 23(6), 535-551.
Clear Purpose Make sure you have your resources available Provide criteria for selection/specific
instructions to guide selection of evidence Require reflection Set clear expectations As to how much evidence (remember lean is good) Clear timeline Clearly communicate to other faculty their role Make sure that feedback is timely
What incentives are there for the learners to complete this portfolio? (there have to be teeth in the process for most learners)
What barriers do you anticipate for: Learners Faculty
Suggestions to overcome barriers…
Introduction Portfolio vocabulary Examples of Individual Development Plans
in electronic portfolios How to create a Individual Development
Plan in an electronic portfolio platform Are they worth the effort?
Are learning portfolios worth the effort? No Its versatility makes it hard to judge whether a portfolio is
good or bad Limited evidence; much obtained from self-report Not clear that effort is rewarded Benefit may be in a side-effect: forces students to write something and teachers to spend some
time with students
Norman, G. BMJ 2008:3337:a514
Are learning portfolios worth the effort? Yes
Portfolio is best solution since it allows the collation and integration of evidence
Portfolios can guide and coach professional development
Flexibility is advantage when carefully developed Mentoring is the single most decisive success
factors Portfolios must be smart and lean Careful implementation is crucialDriessen, E. BMJ 2008:337:a513
Office of Medical Education UCSF School of Medicine
Daniel C. West, MD Patricia O’Sullivan, EdD Lorriana Leard, MD Louise Aronson, MD Chandler Mayfield
Tochel C, Haig A, Hesketh A, Cadzow A, Beggs K, Colthart I, Peackock H. The Effectiveness of Portfolios for Post-Graduate Assessment and Education: BEME Guide No. 12. Med Teach. 2009;31:299-318.
Colbert CY, Ownby AR, Butler PM. A review of portfolio use in residency programs and considerations before implementation. Teach Learn Med. 2008 Oct-Dec;20(4):340-5.
Carraccio C, Englander R. Evaluating competence using a portfolio: a literature review and web-based application to the ACGME competencies. Teach Learn Med. 2004 Fall;16(4):381-7.
Norman G. and Driessen E. Are Portfolios Worth the Effort. BMJ 2008; 337:a514