CCPTP conference: 2010 Joint Conference

Michael J. Scheel, Ph.D.University of Nebraska-Lincoln

Portfolio Evaluation: Assessing Competency

Portfolio: The Concept“A purposeful collection of student work that tells

the story of the student’s efforts, progress, or achievement in a given area(s)” (p. 36; Arter & Spandel, 1992).

Portfolio assessment is used for faculty tenure evaluations; for some license or diplomate evaluations

In counseling, beyond performance on a standardized test or completion of courses, case examples serve to demonstrate competence (Coleman, 1996).

“An artist’s portfolio shows not only what the artist knows but how he or she can put that knowledge into practice over a range of situations” (p. 219; Coleman, 1996).

Portfolio: The AdvantagesCan demonstrate what one knows that reflects the

complexity of the topic and the integration of skills with knowledge (Collins, 1992).

Integration of information collected over several contexts

Provides stimulus for discussion in which the student can present the meaning of an activity or a competency.

Can demonstrate increasing sophistication over time

Stimulates self-reflection because the learner is responsible for deciding the goal and the content of the assessment materials.

The portfolio process1) Determine the goal2) Identify what evidence will demonstrate

acquistion of the goal3) Submit a caption (competence statement)

about the importance of the evidence4) Prepare a statement (narrative) about

how the portfolio as a whole reflects learning

5) Fosters a self-reflective process by the learner (Collins, 1992)

Examples of evidence/artifactsClient rating of counselor effectiveness; supervisor

evaluationsSelf-evaluations; integrative narratives of a content areaVideotapes of therapy sessions that demonstrate an

intervention approachCase presentations; Ethical dilemmas presented through a

caseA personal ethical decision-making modelPersonal theory of changeOne page essays on theory orientation, multicultural

competence, use of assessment in practice etc. similar to an APPIC app.

Case presentations and write-upsOutreach programs presented with powerpoints;

presentations at prof. conferences; publicationsPsych. assessments conducted; written reviews of

assessment instruments; reviews of literature

UNL Copsych PortfolioOnly the counseling psychology program

within the Dept. of Ed. Psych. uses portfolio evaluation with an oral examination; satisfies the Graduate College comprehensive exam requirement

Continuance refining process since 2004; Copsych program approved latest version in the Fall of 2009 as the only comprehensive exam method for our program

UNL Portfolio Procedure1) meet with your academic advisor to discuss the

portfolio procedure and plans for completing comprehensive examinations

2) develop a plan during Program of Study committee meeting in consultation with your doctoral committee regarding the portfolio procedure; doctoral committee members are invited to participate; all program core faculty are required to participate

3) work closely with your academic advisor to assemble portfolio; submit a completed version to your advisor; portfolio is comprised of a narrative summary for each competency area; written narratives are approx. 7 to 10 pp. for each area; the bulk of the portfolio is comprised of these narrative summaries; a limited number of accompanying documents/artifacts may be included as supplementary materials to enhance or further illustrate information contained in the narrative summaries; E.g., theoretical orientation essay; mc self-exploration paper, integrated assessment report.

UNL Portfolio Procedure (cont.)4) Advisor distributes portfolio to each committee

member and each committee member reviews5) Meeting is held with the committee and student to

discuss each area of the portfolio; Committee members bring questions and needs for clarification to the meeting after individual reviews of the portfolio

6)Committee arrives at a consensus decision of ‘pass’ or ‘no pass’ after the conclusion of the meeting;

7)If ‘no pass’ is assigned to a portion of the exam, the student will be given instructions concerning changes and additions to be required to bring the area or areas up to a passing level.

Contextual factors1) The majority of coursework that fulfills the

Counseling Psychology Core Curriculum should be completed at the time of Comprehensive Exam (most students complete portfolio exam in 5th or 6th semester).

2) Doctoral students applying for internship require notification from the Director of Training that they have completed their comprehensive exam at the time of their application. Thus, successful completion of comprehensive examinations (as well as successful proposal of dissertation) must be achieved by October 15th of the year in which the student is applying for pre-doctoral internship.

Portfolio ContentsScoring rubrics are provided to assist students in

considering what material to include in their doctoral portfolios. Specifically, students are expected to demonstrate competence in the following six areas:

1. Measurement, Assessment, and Psychological Testing

2. Counseling Theories and Practice3. Career Development4. Multicultural Issues5. Ethical, Legal and Professional Issues6. Area of Emphasis

Scoring Rubric

Counseling Theory and PracticePortfolio: Examples of performance

indicators1) Knows constructs and major theories and

systems of thought2) Personal theory of change articulated3) Counseling interventions 4) Evaluate effectiveness of treatment

Foundational competencies that line up with portfolio performance indicatorsFoundational Comp Benchmarks: - Scientific knowledge/methods: applies evidence

in practice; compares and contrasts theoretical perspectives; scientific mindedness – articulates issues derived from lit. in supervision and case conf; reviews scholarly work related to clinical practice; willingness to present work for scrutiny by others

- Reflective practice self-assessment: awareness of personal strengths; summarizes lit relevant to client care

- Individual and cultural diversity: monitors and applies knowledge of self as a cultural being in assessment, treatment, and consultation

- Interdisciplinary Systems: demonstrates ability to articulate the role that others provide in service to clients

Functional CompetenciesAssessment, Diagnosis, and Case

Conceptualization: identifies and conceptualizes symptoms through a developmental context; formulates treatment plans

Intervention: Knowledge of interventionsIntervention: planningIntervention: implementation - applies evidence-

based interventions that take into account empirical support, clinical judgment, and client diversity; clinical skills used in development of therapeutic relationships; uses clinical judgment effectively

Portfolio area: Multicultural CounselingExamples of performance indicators1)Integrated conceptualization of mc

competencies2)Awareness of personal values other

cultural groups3)Knowledge and application of mc theories

of psychology4)Demonstrates mc skills5)Demonstrated plan for continued mc

competence

Foundational Competencies: Multicultural Counseling

- Reflective practice self assessment: reflective practice self-identifies cultural identities; articulates attitudes, values, and beliefs toward diverse others

- Relationships: interpersonal relationships: negotiates relationships with ind and groups that significantly differ from oneself

- Individual Cultural Diversity: Self-awareness (e.g., knowledge of self as a cultural being; understands cultural identities; critically evaluates feedback and initiates consultation/supervision over diversity) Applied knowledge (e.g., knowledge of cultural diversity science, theory, and contextual issues; sensitivity to treatment applications with diverse others; understands individual cultural diversity and APA policies; conceptualizes diversity in appropriate and effective ways; considers diversity in assessment, diagnosis and case conceptualization; uses culturally sensitive instruments; seeks consultation; knowledge of multiple identities in case presentations and impact on these treatments)

Functional CompetenciesAssessment, diagnosis and case

conceptualization: assessment (e.g., aware and use culturally sensitive instruments); Integration (e.g., displays know. of assessment tools specific to a client pop. & specific site)

Intervention: Planning (e.g., contexually oriented including diversity); Implementation (e.g., applies evidence interventions while considering cultural diversity)

Supervision and teaching: Awareness of factors affecting quality (e.g., impact of diversity on prof. settings and supervision; awareness of APA and other policies as interpreted through cultural and other contexts)

Future directions for portfolio evaluation developmentFurther lining up of portfolio criteria with the

competency benchmarksA need for research to validate portfolio

assessment as a method to be used to measure competence (e.g., concurrent validity studies)

Need to develop reliable methods of scoring and evaluation

Any scoring method must recognize and value the holistic and integrated nature of the performance (Collins, 1992). Resist the temptation to be reductionistic.

References Baltimore, M. L., Hickson, J., George, J. D., Crutchfield, L. B.

(1996). Portfolio assessment: A model for counselor education. Counselor Education and Supervision, 36(2), 113-121.

Coleman, H. L. K. (1996). Portfolio assessment of multicultural counseling competency. The Counseling Psychologist, 24(2), 216-229.

Coleman, H. L. K. (1997). Multicultural counseling competencies: Assessment, Education, and Training, and Supervision. In Pope-Davis, D. B. & Hardin, L. K. (Eds.) Multicultural aspects of counseling series, Vol. 7. pp. 43-59. Thousand Oaks, CA: Sage.

 Collins, A. (1992). Portfolios for science education: Issues in purpose, structure, and authenticity. Science Education, 76(4), 451-463.

Van der Schaaf, M. F., & Stokking, K. M. (2008). Developing and validating a design for teacher portfolio assessment. Assessment and Education in Higher Education, 33(3), 245-262.

Email address for UNL portfolioTheodore.bartholomew@huskers.unl.edumscheel2@unl.eduFor requests of copies of the portfolio