2010 spa annual meeting san jose, california cross-informant assessment: concepts, behavioral...

2010 SPA Annual MeetingSan Jose, California

Cross-Informant Assessment: Concepts, Behavioral Science, and Applications

Marvin W. Acklin, Ph.D.Honolulu, Hawaii

Joshua Dwire, PsyDGrenada, The Caribbean


Broad focus of this symposium is the use of others in personality assessment

Using how people see each other as a source of clinical information

Looking at the relationship between how people are seen by themselves and how they are see by others

Focus on applied clinical science


Data Sources in Psychological Assessment

Current views of psychological assessment endorse the use of multiple information sources.

Psychological assessment is concerned with a variety of test scores, generally obtained from multiple test methods, considered in the context of history, referral information, and observed behavior: unstructured & structured interviews; performance-based cognitive & personality tests, & observer rating scales (Meyer, et al, 2001)

Each information source has both strengths and constraints in the assessment procedure.


Reliance on Self-Report

Traditional diagnostic practices rely heavily on individual self report as the main source of information in clinical assessment.

There are serious limitations in the validity of self-report, not the least of which is the patient’s motivation to communicate frankly (self-presentational bias).


Use of Observer-Informant Information

The disciplined, empirically validated use of informants is both recent and innovative. The use of informants is most effective using parallel forms of items for behavior ratings.

The Achenbach System of Empirically Based Assessment (ASEBA) is a life-span approach using self-report and Informant ratings is one highly effective method of obtaining observer ratings.

I see you (observer rating) I see you, you see me (Criss-cross) I see you seeing me (metaperception; reflected appraisals)


Limitations of Self-Report

Self report can be limited by defensive and self-presentational biases (Westen & Weinberger, 2004).

“…people have minimal access to many of their cognitive processes, and they often confabulate explanations for their behavior by applying intuitive attributional theories…

Validity Scales are utilized to correct for self-presentational bias.

Critique of validity scales as a basis for correcting self reporting bias (Piedmont et al, 2000).


In order to overcome the drawback of solely using self-reports in psychological assessment, Achenbach, Dumenci, and Rescorla (2005) suggested that cross-informed methods provide a multifaceted approach to assessment in which constructs for evaluation come from the same pool of items.


Concordance of Information Sources: Cross-Method and Cross-Informant

Research demonstrates surprisingly low levels of concordance between different information sources (Meyer, et al, 2001; Meyer, 2002).

“The substantial independence between sources of information is not a phenomenon that is limited to rarified psychological research on personality…” (Meyer, 2002, p. 75).

“…Campbell and Fiske (1959) noted how relative independence among psychological methods can point to unappreciated complexity in the phenomena under investigation.”


Concordance of Information Sources: Cross-Informants (Adults)

Research demonstrates surprisingly low levels of concordance between self and other report and concordance in observer reports:

.681 substance abuse

.428 internalizing disorders

.438 externalizing disorders

.304 mean

.23 self vs. clinician

.26 self vs. aggregated peer rating

.29 self vs. spouse


Adults

Self vs. spouse: Personality & mood .29 Self vs. spouse/partner: Big five .44 Self vs. peer: Personality & mood .27 Self versus peer: Big Five Factors .31 Self vs. significant other: Axes II PD diagnosis .21 Self vs. clinician: Axis II PD traits .33 Self vs. clinician: Big Five Factors .32 Self vs. cognitive test: Memory problems .13


Children

Research also demonstrates surprisingly low levels of concordance between self and other report and concordance in observer reports ofnchildren:

.60 similar informants (e.g., parents)

.28 different informants

.22 self and other ratings


Children

Self vs. parent: Behavioral & emotional problems .29 Self vs. clinician: Behavioral & emotional problems .14 Self vs. teacher: Behavioral & emotional problems .21 Parent vs. direct observer of child behavior: .27 Low concordance suggests situational specificity in informant

ratings and the fundamental independence of ratings.


One approach to capitalizing on the potential added value of ratings by multiple informants is known as “360° feedback” methodology. This methodology entails comparing assessment data obtained from ratings by multiple informants, as well as from self-ratings. Many businesses have adopted 360o feedback for assessing managers' job performance by having subordinates, peers, and bosses rate the managers' performance for comparison with the managers' self-ratings (Brett & Atwater, 2001).


Self-ratings had mean correlations of.19 with peer ratings and.22 with supervisor ratings. And between supervisors and peers, the mean correlation was.34. Despite the relatively small correlations, a meta-analysis of the incremental validity of subordinate and peer ratings led Conway, Lombardo, and Sanders (2001) to conclude that the “results provided justification for the added expense of including subordinates and peers in a rating system”


Cross-Informant Assessment of Personality Disorders

Personality Disorders: “…diagnoses have only moderate associations when they are derived from self-reports or the reports of parents, significant others and clinicians

Self and/or Other attributional biases in personality disordered reporters or informants may play a role in further reducing concordance.


Limitations of Observer-Reports

The constraints of observer ratings include parameters of the particular type of relationships (e.g., spouse, coworker, therapist) and the setting in which observations are made.

Appear to be independent sources

Others reports may be systematically biased due to a variety of attributional biases—for example in divorcing spouses.


Limitations of Observer-report

Theoretical implications of cross-method and cross-informant methods.

Weak support for the concept of consensual reality. The low level of concordance between sources, especially between self and observer report threatens simplistic views of consensual reality.

On the other hand, “…using a composite judge can yield very large validity gains when the redundancy of judges is low…” (Tsujimoto, Hamilton, & Berger, 1990)


Use of the Achenbach scales in assessment practiceThe Achenbach System of Empirically Based Assessment comprises a family of forms for assessing adaptive functioning

and problems is easy and cost-effective ways.

Child Behavior Checklist

Teacher-Caregiver Report FormYouth Self-ReportAdult Self Report

Adult Behavior ChecklistOlder Adult Self-Report

Older Adult Behavior Checklist


Introduction to Symposium Papers

Cross-informant assess using the Inventory of Interpersonal Problems (Pincus)

Cross-informed assessment of marital compatibility (Dwire & Acklin)

Self and Informant ratings using the NEO-PIR (Kurtz) Parent personality and infant temperament (Janson

& Narde)

2010 spa annual meeting san jose, california cross-informant assessment: concepts, behavioral...

Documents