assessment of trainee competencies: preliminary psychometric study

Assessment of Trainee Competencies: Preliminary

Psychometric StudyDavid J. Martin, Ph.D.

Harbor-UCLA Medical CenterGeffen School of Medicine at UCLA

Models of Competency

• Cube Model of Competency Development (Rodolfa, Bent, Eisman, Nelson, Rehm & Ritchie, 2005)– Functional Competency– Foundational Competency– Stages of Professional Development


• Cube Model (Rodolfa et al., 2005):– Functional Competency

• Assessment-Diagnosis-Case Conceptualization

• Intervention• Consultation• Research Evaluation• Supervision-Teaching• Management-Administration


• Cube Model (Rodolfa et al., 2005):– Foundational Competency

• Reflective Practice-Self Assessment• Scientific Knowledge-Methods• Relationships• Ethical-Legal Standards—policy• Individual-Cultural Diversity• Interdisciplinary Systems


• Cube Model (Rodolfa et al):– Stages of Professional Development

• Doctoral Education• Doctoral Internship/Residency• Post-doctoral Supervision• Residency/Fellowship• Continuing Competency


• Practicum Competencies Outline – Association of Directors of Psychology

Training Clinics (ADPTC) Practicum Competencies Workgroup (February 27, 2004)

– Council of Chairs of Training Councils (CCTC) Practicum Competencies Workgroup (March 25, 2004)

– Final CCTC comments, November, 2004– Summarized by Hatcher & Lassiter (2006)


• Practicum Competencies Outline– Levels of Competence

•Novice• Intermediate•Advanced•Proficient•Expert


• Practicum Competencies Outline– Areas of Competence

• Relationship/Interpersonal Skills– With Patients/Clients/Families– With Colleagues– With Supervisors– With Support Staff– With Teams at Clinic– With Community Professionals– For the Practicum Site Itself



• Skills in Application of Research• Intervention Skills• Psychological Assessment Skills• Consultation Skills/Interdisciplinary• Collaborations• Diversity (Individual & Cultural

Differences)



• Ethics• Professional Development

– Practical Skills to Maintain Effective Clinical Practice

– Professional Development Competencies

• Metaknowledge/Metacompetencies• Development of Leadership Skills• Supervisory Skills

Harbor-UCLA Assessment

• Psychological assessment skills

Extern Fellow

Ability to select and implement multiple methods and means of evaluation in ways that are responsive to and respectful of diverse individuals, couples, families, and groups

I P

Ability to utilize systematic approaches to gathering data to inform clinical decision making

I P

Knowledge of psychometric issues and bases of assessment methods

A P

Knowledge of issues related to integration of different data sources

A P

Harbor-UCLA Assessment

Competency Level

Unsatisfactory Needs Improvement

Satisfactory Strong Outstanding N/A

1. Ability to select and implement multiple methods and means of evaluation in ways that are responsive to and respectful of diverse individual, couples, families and groups.

P 1 2 3 4 5 6

2. Ability to utilize systematic approaches to gathering data to inform clinical decision making.

P 1 2 3 4 5 6

3. Knowledge of psychometric issues and bases of assessment methods.

P 1 2 3 4 5 6

4. Knowledge of issues related to integration of different data sources.

P 1 2 3 4 5 6

Warning !

Attempts at complete review of the following slides may result in

serious information overload!

Internal ConsistencyFellows Externs

Items N α N α

Relationship/Interpersonal Skills

With Patients/Clients/Families 4 90 .938 203 .918

With Colleagues 3 85 .923 186 .902

With Supervisors 6 92 .958 206 .948

With Support Staff N/A N/A N/A N/A N/A

With Teams at Clinic 2 77 .907 184 .900

With Community Professionals N/A N/A N/A N/A N/A

For the Practicum Site Itself 3 74 .935 178 .928

Internal ConsistencyFellows Externs

Items N α N α

Skills in Application of Research 2 87 .910

-- --

Intervention Skills 8 70 .954

98 .959

Psychological Assessment Skills 7 55 .960

92 .950

Consultation Skills/Interdisciplinary Collaborations

7 60 .961

110 .933

Diversity (Individual and Cultural Differences)

3 91 .955

200 .939

Ethics 6 86 .972

188 .975

Professional Development

Practical Skills to Maintain Effective Clinical Practice

6 75 .938

161 .943

Professional Development Competencies 11 73 .964

136 .964

Metaknowledge/Metacompetencies 5 91 .914

190 .939

Development of Leadership Skills 8 24 .966

-- --

Supervisory Skills 5 56 .901

-- --

Descriptive Statistics

FellowsN Min Max Mean S.D.


With Patients/Clients/Families 90 3.00 5.00 4.55 .609

With Colleagues 85 3.00 5.00 4.55 .619

With Supervisors 92 3.00 5.00 4.55 .622

With Support Staff 86 3.00 5.00 4.67 .583

With Teams at Clinic 77 3.00 5.00 4.55 .650

With Community Professionals 76 3.00 5.00 4.47 .739

For the Practicum Site Itself 74 3.00 5.00 4.53 .682

Descriptive StatisticsFellows

N Min Max Mean S.D.

Skills in Application of Research 87 3.00

5.00 4.30 .688

Intervention Skills 70 3.00

5.00 4.47 .612

Psychological Assessment Skills 55 3.00

5.00 4.32 .669


60 3.00

5.00 4.42 .654


91 3.00

5.00 4.49 .639

Ethics 86 3.00

5.00 4.52 .640



75 2.17

5.00 4.38 .700

Professional Development Competencies

73 3.00

5.00 4.50 .590

Metaknowledge/Metacompetencies 91 3.00

5.00 4.48 .580

Development of Leadership Skills 24 3.00

5.00 3.96 .637

Supervisory Skills 56 3.00

5.00 4.16 .570

Descriptive Statistics

ExternsN Min Max Mean S.D.


With Patients/Clients/Families 203 2.75 5.00 4.20 .656

With Colleagues 186 3.00 5.00 4.28 .639

With Supervisors 206 1.50 5.00 4.23 .711

With Support Staff -- -- -- -- --

With Teams at Clinic 184 1.50 5.00 4.17 .761

With Community Professionals 153 2.00 5.00 4. 14 .776

For the Practicum Site Itself 178 2.67 5.00 4.14 .732

Descriptive StatisticsExterns

N Min Max Mean S.D.

Skills in Application of Research -- -- -- -- --

Intervention Skills 98 2.13

5.00 3.82 .731

Psychological Assessment Skills 92 3.00

5.00 4.21 .760


110 3.00

5.00 4.21 .597


200 2.00

5.00 4.13 .740

Ethics 188 1.50

5.00 4.25 .739



161 2.17

5.00 4.02 .748


136 2.18

5.00 4.24 .675

Metaknowledge/Metacompetencies 190 2.00

5.00 4.15 .682

Development of Leadership Skills -- -- -- -- --

Supervisory Skills -- -- -- -- --

Inter-Scale Correlations

• Fellows’ Inter-Scale Correlations Ranged From .39 (Relationship Skills with Support Staff – Leadership Skills) - .93 (Assessment Skills – Consultation Skills)

• Externs’ Inter-Scale Correlations Ranged From .63 (Relationship Skills with Team – Ethics) - .95 (Intervention Skills – Consultation Skills)

Pre – Post Comparisons

FellowsEvaluation Period t p

1 2

N M S.D. N M S.D.


With Patients/Clients/Families 42 4.37 .723 48 4.71 .438 2.77 .007

With Colleagues 40 4.33 .693 45 4.74 .471 3.26 .002

With Supervisors 44 4.39 .698 48 4.69 .511 2.34 .02

With Support Staff 42 4.52 .707 44 4.82 .390 2.41 .018

With Teams at Clinic 38 4.41 .761 39 4.68 .493 1.86 .066

With Community Professionals 35 4.14 .879 41 4.76 .435 3.94 .000

For the Practicum Site Itself 38 4.32 .803 36 4.75 .439 2.80 .006

Pre – Post ComparisonsFellows

Evaluation Period t p

1 2

N M S.D. N M S.D.

Skills in Application of Research

41 4.13 .775 46 4.46 .566 2.23 .028

Intervention Skills 32 4.20 .687 38 4.70 .431 3.72 .000

Psychological Assessment Skills

25 .410 .803 30 4.50 .474 2.31 .025


32 4.18 .798 34 4.64 .411 2.93 .005


44 4.32 .753 47 4.65 .463 2.51 .014

Ethics 40 4.33 .727 46 4.68 .479 2.68 .009



32 4.08 .754 43 4.59 .574 3.33 .001


30 4.23 .691 43 4.69 .421 3.55 .001

Metaknowledge/Metacompetencies

46 4.33 .656 45 4.62 .476 2.37 .020

Development of Leadership Skills

13 3.65 .545 11 4.32 .560 2.94 .008

Supervisory Skills 28 3.92 .602 28 4.41 .420 3.50 .001

Pre – Post Comparisons

ExternsEvaluation Period t p

1 2

N M S.D. N M S.D.


With Patients/Clients/Families 99 4.01 .674 104

4.38 .587 4.20 .000

With Colleagues 90 4.13 .591 96 4.41 .656 3.05 .003

With Supervisors 101 4.07 .710 105

4.38 .683 3.14 .002

With Support Staff 88 4.18 .736 93 4.51 .653 3.13 .002

With Teams at Clinic 89 3.99 .748 95 4.34 .738 3.17 .002

With Community Professionals 66 4.00 .744 87 4.26 .784 2.11 .036

For the Practicum Site Itself 89 3.97 .757 89 4.31 .665 3.26 .001

Pre – Post ComparisonsExterns

Evaluation Period t p

1 2

N M S.D. N M S.D.

Skills in Application of Research

-- -- -- -- -- -- -- --

Intervention Skills 49 3.57 .665 49 4.06 .715 3.56 .001

Psychological Assessment Skills

44 4.56 .811 48 4.92 .674 2.27 .026


51 4.08 .645 59 4.32 .532 2.15 .034


97 3.98 .771 103 4.28 .683 2.87 .004

Ethics 91 4.08 .763 97 4.42 .680 3.20 .002



77 3.86 .721 84 4.17 .743 2.74 .007


68 4.11 .676 68 4.376

.651 2.34 .021

Metaknowledge/Metacompetencies

91 4.01 .651 99 4.271

.651 2.64 .009

Development of Leadership Skills

-- -- -- -- -- -- -- --

Supervisory Skills -- -- -- -- -- -- -- --

Discussion

• Preliminary data suggest that competencies can be measured reliably

• Inter-scale correlations raise important questions concerning competency constructs

Discussion

• Future Directions– Need for additional data to inform

competency evaluation process– More controlled study of competency

scales (e.g., mid-term and final ratings)– Factor-analytic approaches may address

questions of inter-scale shared variance– Need for validation strategies

assessment of trainee competencies: preliminary psychometric study

Documents

relationship skills

final cctc comments

additional data

final rating

council of chairs

serious information

important questions

association of directors