TEMPLATE DESIGN © 2008

www.PosterPresentations.com

Curriculum Wide Student Portfolios Documenting Progression and Achievement of Curricular Outcomes: A Pilot ProjectPamela L. Stamm, Kristen L. Helms, Vishnu Suppiramaniam, Sarah Anne Swann, Laura J. Van Deventer, Jordan Garrison, Selena Day

INTRODUCTION RESULTSMETHODS RESULTS

• The current portfolio proposal allows for documentation of student’s perceived growth and achievement of curricular outcomes through their reporting, uploading of artifacts mapped to outcomes, and self-assessment of achievement (Not Encountered, Needs Development, and Competent) and level of performance (Knows, Knows How, Shows How, and Shows) of each task or outcome. Students will need further guidance to insure this is done through their reporting.

• A limitation of the current proposal is that students will self assess achievement and knowledge on task descriptions once and will later only do this for the overall outcome. This data will be difficult to correlate. Possible solutions include:

• Aligning the self-assessments so that students self –assess the overall outcomes and/or the task descriptions on more than one occasion.

• Correlating initial task description self-assessments to the self-assessment on task descriptions obtained from the Teaching Effectiveness Survey conducted at the end of the P4 year.

• As currently proposed, the time commitment for both students appears close to the targets of 60 minutes for reports on task descriptions and 30 minutes for reports on outcomes. The pilot is only an estimate of the time commitment so this will need to be monitored as the portfolio is implemented

• The committee is also proposing that assignments that overlap with the proposed portfolio coursework be eliminated

• The time commitment for evaluation by aculty was also considered acceptable• The majority of progress reports were not acceptable to the reviewers. The

committee has since been reviewing progress reports and confirming this. Some proposed solutions are as follows:• Provide additional orientation regarding reporting format• Providing the rubric in advance to help communicate expectations

• Add a statement to the rubric regarding completion of all questions • Provide and review examples / excerpts of reports with exemplary, competent, and

poor ratings• Feelings are mixed in regards to reporting on Outcome versus Task Descriptions for the

P3 students. Although the “outcome” reports were inadequate, so were the “task description” based reports. It is unclear if implementing the steps above will address the inadequacies of the “outcome” reports

• The proposed portfolio appears to provide data on student’s perceptions of the experienced curriculum and can be used to map the experienced curriculum to the current outcomes

• The attempt to have students map artifacts to outcomes was unsuccessful. This was more likely due to a need for more clarity when instructing students to complete the form in MyFolio

1. Accreditation Council for Pharmacy Education. Accreditation Standards and Guidelines for the Professional Program in Pharmacy Leading to the Doctor of Pharmacy Degree. Available at http://www.acpe-accredit.org/standards/default.asp Accessed July 5, 2011.

2. Zubizaretta, John. The Learning Portfolio. San Francisco: John Wiley & Sons, 2009.

• Portfolios stimulate deep thoughts and can be designed to display best works or learning, to serve as personal reflective notebooks or to evaluative purposes

• Spring 2010, an HSOP Ad Hoc Committee consisting of students, staff, and faculty from all departments was formed to develop a student portfolio proposal

• The committee reviewed the following background material• Accreditation Council for Pharmacy Education (ACPE) Accreditation Standards and

Guidelines for the Professional Program in Pharmacy Leading to the Doctor of Pharmacy Degree

• The Learning Portfolio by John Zubazaretta• Guideline 15.1 states: “that the college or school’s evaluation of student learning

should…..demonstrate and document in student portfolios that graduates have attained the desired competencies, when measured in a variety of health care settings”

• Guideline 15.4 states: “Student portfolios should be employed to document students’ progressive achievement of the competencies throughout the curriculum and the practice experiences. The portfolios should be standardized and include student self-assessment, as well as faculty and preceptor assessments of the educational outcomes.”

• Thus the committee proposed the following objectives. The portfolio should:1. Document deep and thoughtful learning through the process of reflection,

documentation, and mentoring2. Demonstrate a deeper understanding of the HSOP curricular outcomes (Appendix

A) and the link between coursework and those outcomes3. Show progression toward and achievement of curricular outcomes (ACPE

Standards Appendix B)4. Highlight student pharmacists’ best works that would be of interest to future

employers. • The committee proposed the following five portfolio sections.

1. Curriculum Vitae 4. Professionalism Self Assessments

2. Long Term and Short Term Goals 5. Artifacts / Best Works3. Curricular Outcomes Self Assessments

• The pilot project focused on Curricular Outcomes Self Assessment section • General layout in MyFolio

• Main level folder: called Curricular Outcomes Folder• Eight subfolders: one for each of the HSOP’s eight curricular outcomes• Each subfolder contains tables for the following items:

• Outcome• Task descriptions: descriptions of skills associated with each outcome• Time tracking• Artifacts that support their report

• Students will complete a form in MyFolio providing the following data:1. Evaluate their level of achievement on the task description or outcome

• Not encountered, Needs Development, Competent2. Self assess their ability to perform the task description or outcome

• Knows, Shows How, Shows, Does3. Report on their progress toward achieving the task description or outcome4. Identify the courses that helped them achieve the task description or outcome 5. Upload artifacts that would support their report6. Map the artifact to the Outcome’s task descriptions7. Identify the time needed to complete report

OBJECTIVES

Figure 2. First year pharmacy students stating that the course helped them progress toward achieving the task description (n=28)

Table 1. Due dates and Portfolio Evaluation datesProfessional Year / Semester

Progress due(# outcomes)

Outcomes / Task Descriptions Due

Reviewed by faculty mentor

P1 Spring Spring mentor meeting 6 and 8 Spring mentor meetingP2 FallP2 Spring

Start of fall semester Spring mentor meeting

1 and 32 and 4

Fall mentor meetingSpring mentor meeting

P3 FallP3 Spring

Start of fall semester Spring mentor meeting

End of finals

5 and 75 – 8 1 – 4

Fall mentor meetingSpring mentor meeting

End of semester

P4 year TBD TBD TBD

• The committee had several concerns • Added student workload• Effectiveness of the section design, use of MyFolio, and workload.

• To address these the committee conducted a pilot project with objectives as follows:1. Develop a curriculum wide student portfolio documenting progression and

achievement of curricular outcomes 2. Assess workload of the Curricular Outcomes section of the proposed portfolio 3. Evaluate the effectiveness of current guided questions 4. Appraise the quality of task versus outcomes oriented reflections5. Assess the ability to compare the designed and experienced curriculum

Figure 1. Screenshot of portfolio form in E*Value MyFolio

• The pilot was held April 8, 2011 during the Professional Seminar Series, a course for all students in the first through third professional years with required attendance

• Students attend in person in 1 large auditorium and via video conference equipment in small group rooms on campus as well as in 2 auditoriums at a satellite campus

• The committee chair and a student provided an overview of the portfolio (Appendix C)• A committee member was in attendance in each of the video conferenced rooms to

facilitate questions • Each student was assigned a specific outcome with each outcome assigned to 14-21

students from each of the three professional years• The intent was for students to begin completing their report during the class; however,

E*Value could not support that number of simultaneous users logging into the system so detailed instructions including MyFolio screenshots were emailed to each student

• Students had 10 days to complete their report• P1 and P2 students completed reports on each task description for their assigned

outcome• P3 students completed a report on the overall outcome only• Students were expected to name documents they would have loaded as artifacts

rather than uploading artifacts• MyFolio reports were generated and downloaded into Microsoft Excel 2010 for review

and descriptive analysis• Student time tracking was used to determine the median (range) time needed to

complete the report by class and outcome. A median time of • Two faculty reviewed the reports on progress piloting a proposed rubric (Appendix D).

The rubric has two dimensions: 1) Reflections and 2) Artifacts. Since artifacts were not uploaded, reports were only assessed on the first dimension.

• A stopwatch was used to assess time needed to review the reports. The longer of the two times was recorded in order to estimate time commitment for faculty when assessing the portfolio. Since faculty generally grade independently and since the faculty will need to be trained on the rubric to improve inter-rater reliability, time for discussion of the report and rubric was not included in the time estimate

Table 2. Reflection Domain of the Proposed Evaluation Rubric

Needs Development Competent Exemplary

REFLECTI

ONS

1. Ideas not related to artifacts or supported by written evidence

2. Content does not describe current skill or knowledge level

3. Content does not clearly communicate depth or breadth of experiences (superficial or rambling)

Writing mechanics are of unacceptable quality (i.e., frequent errors in writing mechanics: verb subject agreement, tense, spelling errors)

1. Ideas are supported with written evidence or related to artifacts, but reader must make those connections

2. Content describes current skill or knowledge level

3. Content usually reveals depth and breadth of experiences (Some reflections are limited in detail; others reveal depth and breadth of experiences)

4. Writing mechanics are of acceptable quality (i.e., occasional errors in writing mechanics: verb subject agreement, tense, spelling errors)

1. Ideas are clearly supported with written evidence or clearly related to artifacts

2. Content describe growth in skill or knowledge over time

3. Content consistently reveals depth and breadth of experiences

4. Writing mechanics are of excellent quality (i.e., grammar, punctuation, and spelling)

Submission is Unacceptable: Missing reports Missed deadline

IMPLICATIONS & FUTURE DIRECTIONS

REFERENCES

P1 P2 P30%

10%20%30%40%50%60%70%80%90%

100%

12 11 11

6 5 6

1 2 1

Needs Development Competent Exemplary

Professional Year / Semester

Outcomes / Task Descriptions Due

Median time Interquartile Range

P1 Spring 6 and 8 90 64 – 120 P2 FallP2 Spring

1 and 32 and 4

90105

65 – 120 75 – 165

P3 FallP3 Spring

5 and 75 – 81 - 4

90128158

75 – 120 109 – 94 176 – 180

P4 year TBD TBD TBF

Median time (25-75 percentile) in minutes

Outcome (n) P1 P2 P31 (23,0,10) 45 (30-68) NA 30 (19-30)2 (18,19,19) 45 (30-60) 60 (45-90) 60 (45-90)3 (19,14,7) 30 (15-45) 53 (34-60) 30 (15-38)4 (20,17,14) 38 (30-60) 45 (30-75) 38 (30-56)5 (15,13,14) 30 (23-60) 60 (45-60) 30 (19-45)6 (18,18,14) 45 (34-60) 45 (30-71) 30 (30-45)7 (16,17,10) 45 (30-60) 30 (30-60) 38 (30-56)8 (17,22,9) 45 (30-60) 60 (30-90) 30 (30-30)

Overall Median (145, 120, 96)

45 (30-60) 45 (30-75) 30 (30-60)

Overall Range(145, 120, 96)

15-120 15-150 15-150

Table 3. Median (Interquartile ranges) time spent on reporting of task description (P1’s and P2’s) and Outcomes (P3’s)

Table 4. Estimated student time commitment per due date according to current proposed timeline.

Number (%)

CAPP Drug Lit

Pharmacokinetics

PPE

Form a covenantal relationship with patient, motivated by care and respect

-- 13 (46) 1 (4) 22 (79)

Assess patient understanding of illness and treatment (health literacy)

16 (57) 2 (7) 1 (4) 13 (46)

Prioritize / triage patient problems 11 (39) 1 (4) 2 (7) 11 (39)

Evaluate patient specific drug therapy and non-drug therapy

10 (36) 3 (11) 4 (14) 14 (50)

Gather compile and evaluate subjective and objective data (see evaluate patient specific…therapy, above)

17 (61) 6 (21) - 17 (61)

Assess patient readiness / motivation / ability to accept therapy / lifestyle recommendations

13 (46) -- 1 (4) 13 (46)

Determine pharmacotherapy goals (patient advocacy, collaboration with patient and other healthcare professionals)

11 (39) 2 (7) 2 (7) 15 (54)

Figure 2. Rubric Scores for Individual Progress Reports on Task Descriptions or Outcomes for Outcome 2 (n=18 per class)