1 measuring group-level psychological properties (a tribute to larry james) daniel a. newman...
TRANSCRIPT
![Page 1: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/1.jpg)
1
Measuring Group-Level Psychological Properties
(A Tribute to Larry James)
Daniel A. Newman
University of Illinois
Daniel A. Newman, Ph.D.
![Page 2: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/2.jpg)
22
OverviewGroup-Level Psychological Properties?
1.Psychological Climate Group-Level vs. Individual-Level Constructs
2.Aggregation Bias
3.Why we need rWG (Within-group agreement) Justifying Aggregation
4.rWG(J) for multi-item scales Agreement vs. Reliability
![Page 3: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/3.jpg)
333
OverviewGroup-Level Psychological Properties?
1.Psychological Climate James & Jones (1974), Jones & James (1979), James & Sells (1981),
James (1982), James et al. (1988), James & James (1989)
2.Aggregation Bias James (1982), James et al. (1980)
3.Why we need rWG (Within-group agreement) James (1982), James, Demaree, & Wolf (1984; 1993), George &
James (1993)
4.rWG(J) for multi-item scales James, Demaree, & Wolf (1984), LeBreton, James, & Lindell (2005)
![Page 4: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/4.jpg)
4444
OverviewGroup-Level Psychological Properties?
1.Psychological Climate James & Jones (1974), Jones & James (1979), James & Sells (1981),
James (1982), James et al. (1988), James & James (1989)
2.Aggregation Bias James (1982), James et al. (1980)
3.Why we need rWG (Within-group agreement) James (1982), James, Demaree, & Wolf (1984; 1993), George &
James (1993)
4.rWG(J) for multi-item scales James, Demaree, & Wolf (1984), LeBreton, James, & Lindell (2005)
![Page 5: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/5.jpg)
555
Quotes & EquationsIn summarizing Larry James’s contributions to Multilevel Theory, I’ll use a two-pronged approach:
1.Quotes
2.Equations
![Page 6: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/6.jpg)
66
Quotes & EquationsIn summarizing Larry James’s contributions to Multilevel Theory, I’ll use a two-pronged approach:
1.Quotes
2.Equations
)(1 22ExWG j
sr
![Page 7: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/7.jpg)
7
Levels of Analysis• In social science, hypothetical constructs
reside at multiple levels of analysis (or levels of aggregation):– National Level: Culture– Organizational Level: Organizational Climate,
CEO personality, Strategy– Team Level: Team efficacy, Norms, Leader style– Individual Level: Attitude, Personality, Job
Performance, Psychological Climate
![Page 8: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/8.jpg)
8
Levels of Analysis
Group
Organizational
Individual
![Page 9: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/9.jpg)
9
Levels of Analysis• Individuals are nested within Groups• Groups are nested within Organizations
• One level can influence another – Group norms influence individual behavior– Individual behaviors aggregate to produce
group/team performance
![Page 10: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/10.jpg)
1010
Psychological Climate• Psychological Climate – ‘the meaning an
individual attaches to a work environment’
• Organizational Climate – ‘the aggregated meaning; i.e., the typical, average, or usual way people in a setting [work environment] describe it’
• Schneider (1981, pp. 4-5), as cited by James (1982)
![Page 11: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/11.jpg)
111111
Psychological Climate• Psychological Climate – individual level
construct
• Organizational Climate – group level construct
![Page 12: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/12.jpg)
121212
Psychological Climate• “… perceptual agreement implies a shared
assignment of psychological meaning, from which it follows that an aggregate (mean) climate score provides the opportunity to describe an environment in psychological terms.”
• “Furthermore, given perceptual agreement, I submit that a climate construct at the aggregate level is defined in precisely the same manner as it is at the individual level.”
• James (1982, p. 221)
![Page 13: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/13.jpg)
1313131313
Psychological ClimateRelationship between organizational climate and
psychological climate:
• PC = psychological climate perception of person in a group
• OC = organizational climate of the group
)(PCaverageOC
![Page 14: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/14.jpg)
141414
Psychological ClimateRelationship between organizational climate and
psychological climate:
• PCpg = psychological climate perception of person p in group g
• OC0g = organizational climate in group g
g
n
ppgg nPCOC
g
1
0
![Page 15: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/15.jpg)
1515
Psychological ClimateRelationship between organizational climate and
psychological climate:
• PCpg = psychological climate perception of person p in group g
• OC0g = organizational climate in group g • upg = deviation of person p’s individual psych.
climate perception from group g’s org. climate
pggpg uOCPC 0
![Page 16: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/16.jpg)
161616
Psychological ClimateJames & Jones (1974), reviewed 3 approaches to conceptualize & measure org. climate:
1)Org.-Level Attribute, Multiple Measures2)Org.-Level Attribute, Perceptual Measures3)Indiv.-Level Attribute, Perceptual Measures*
* Introduced the term, “Psychological Climate”
![Page 17: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/17.jpg)
171717
James & Jones (1974)• “Returning to the perceptual definition of organizational
climate, it would seem that the reliance on perceptual measurement may be interpreted as meaning that organizational climate includes not only descriptions of situational characteristics, but also individual differences in perceptions and attitudes. This is somewhat confusing if one wishes to employ organizational climate as an organizational attribute or main effect, since the use of perceptual measurement introduces variance which is a function of differences between individuals and is not necessarily descriptive of organizations or situations. Therefore, the accuracy and/or consensus of perception must be verified if accumulated perceptual organizational climate measures are used to describe organizational attributes (Guion, 1973).” (p. 1103)
![Page 18: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/18.jpg)
18181818
Jones & James (1979)• “The [conceptual] argument for aggregating perceptually
based climate scores (i.e., psychological climate scores) appears to rest heavily on three basic assumptions: first, that psychological climate scores describe perceived situations; second, that individuals exposed to the same set of situational conditions will describe these conditions in similar ways; and third, that aggregation will emphasize perceptual similarities and minimize individual differences. Based on this logic, it is generally presumed that empirically demonstrated agreement among different perceivers implies that these perceivers have experienced common situational conditions (Guion, 1973; Insel & Moos, 1974; James & Jones, 1974; Schneider, 1975a),”
• (p. 206).
![Page 19: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/19.jpg)
19191919
James & Jones (1974)• “Although this school of thought [from Schneider and others]
assumes that situational and individual characteristics interact to produce a third set of perceptual, intervening variables, such an assumption does not mean that perceived climate is different from an individual attribute. Rather, the intervening variables are individual attributes which provide a bridge between the situation and behavior.”
• (p. 1107)
• So … “Psychological Climate” is born!
![Page 20: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/20.jpg)
2020202020
James (1982)• “current thinking in climate suggests that the unit of theory
for climate, including organizational climate, is the individual, and the appropriate unit to select for observation is the individual. This thinking is based on the view that climate involves a set of macro perceptions that reflect how environments are cognitively represented in terms of their psychological meaning and significance to the individual.”
• (p. 219)
• So … measuring organizational climate (an org.-level attribute) involves an individual-level true score (i.e., psychological climate).
![Page 21: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/21.jpg)
212121212121
James et al. (1988)• “Shared assignment of meaning justifies aggregation to a
higher level of analysis (e.g., groups, subsystems, organizations) because it furnishes a way of relating a construct (PC) that is defined and operationalized at one level of analysis (the individual) to another form of the construct at a different level of analysis (e.g., group climate, subsystem climate, OC). Although the unit of analysis for the aggregate psychological variable is the situation (e.g., group, subsystem, organization), the definition and basic unit of theory remains psychological.”
• (p. 130, from Organizations Do Not Cognize)
![Page 22: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/22.jpg)
222222222222
James & James (1989)General PC
Leader Support
Role Stress, Conflict,
Ambiguity
Job Autonomy, Challenge
Group Warmth
& Cooperat.
.85.86 .77 .81
- PC = Cognitive evaluation of work environment- See James & Sells (1981), Jones & James (1979)
![Page 23: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/23.jpg)
23232323232323
Psychological Climate
Psych. Climate
Job Satisfaction
/Affect
- Reciprocal relationship between PC and Job Satis./Affect- James & Tetrick (1986), James & James (1992)
![Page 24: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/24.jpg)
242424
Psychological ClimateSummary:•There is a group-level organizational reality (“the situation”)•That reality is reflected in individual-level, psychological perceptions•The individual-level psychological climate perceptions are a meaningful locus of theory•The individual perceptions can be aggregated to represent a group-level, psychological property [if perceptions are shared]
![Page 25: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/25.jpg)
25
Aggregation Bias
Aggregation – combining micro-level data so it can represent the macro-level (typically, by taking an average of micro-level responses)
• The aggregate of individuals’ scores represents the group-level construct
![Page 26: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/26.jpg)
2626
Levels of Analysis
Group
Organizational
Individual
![Page 27: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/27.jpg)
27
Aggregation• Ecological fallacy – generalizing group-level
(aggregate) results to the individual level– Because we know group collectivism is related to group-
level cooperation, we inaccurately assume individual collectivism is related to individual cooperativeness.
• Atomistic fallacy – generalizing individual-level results to the group (aggregate) level– Because we know indiv. IQ is strongly related to indiv.-
level job performance, we inaccurately assume group IQ is strongly related to group performance.
![Page 28: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/28.jpg)
28
Aggregation
The Truth about Aggregates:
• If the individual-level correlation between X and Y is rindiv. = .3, this does not imply that the group-level correlation between X and Y is rgroup = .3.
• Likewise, if the group-level correlation between X and Y is rgroup = .3, this does not imply that the individual-level correlation between X and Y is rindiv. = .3.
![Page 29: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/29.jpg)
29
AggregationDirection of a correlation (+ or -) can change when we
move from the individual level to the group level. Within-Group Correlation Between-GroupCorrelation
Y
X
![Page 30: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/30.jpg)
30
AggregationExample) Foreign birth & Illiteracy (Robinson, 1950).rindiv. = .12; rgroup(states) = -.53
Within-Group Correlation Between-GroupCorrelation
Y
X
![Page 31: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/31.jpg)
31
AggregationTotal correlation is a combination of the individual-
level correlation and the group-level correlation. Within-Group Correlation Between-GroupCorrelation
Total
Correlation
Y
X
rtotal
rbetween
rwithin
![Page 32: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/32.jpg)
32
Aggregation• Total correlation is a combination of the
individual-level (within) correlation and the group-level (between) correlation.
),( withinbetweentotal rrfr
![Page 33: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/33.jpg)
33
Aggregation• Specifically,
• rtotal = overall X-Y correlation, ignoring group membership
• rbetween = between-groups X-Y correlation• rwithin = within-groups X-Y correlation
• (from ANOVA; DV= X, IV= group) [like R2; variance in X accounted for by group membership, then inflated by the
unreliability of group means; i.e., .]
)1)(1()( 22yxwithinyxbetweentotal rrr
totalbetweenx SSSS2
)2(/)1(2 ICCICCx
![Page 34: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/34.jpg)
34
Aggregation• For example, suppose• rbetween = -.45 = between-groups X-Y correlation• rwithin = .20 = within-groups X-Y correlation
• = .64 (from ANOVA; DV= X, IV= group) • = .81 (from ANOVA; DV= Y, IV= group)Then …
)1)(1()( 22yxwithinyxbetweentotal rrr
2x2y
27.)81.1)(64.1(20.)81.64.(45. totalr
![Page 35: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/35.jpg)
35
Aggregation• For example, suppose• rbetween = -.45 = between-groups X-Y correlation• rwithin = .20 = within-groups X-Y correlation
• = .64 (from ANOVA; DV= X, IV= group) • = .81 (from ANOVA; DV= Y, IV= group)Then …
)1)(1()( 22yxwithinyxbetweentotal rrr
2x2y
27.)81.1)(64.1(20.)81.64.(45. totalr
![Page 36: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/36.jpg)
36
AggregationTotal correlation is a combination of the individual-
level correlation and the group-level correlation. Within-Group Correlation Between-GroupCorrelation
Total
Correlation
Y
X
rtotal
rbetween
rwithin
![Page 37: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/37.jpg)
37
Aggregation
Implications:• Even if total correlation between X and Y
(rtotal) is statistically significant, – rwithin might not be– rbetween might not be
* Many studies in top journals report total relationships between variables, while ignoring nesting/ nonindependence (e.g., different groups, different jobs, different supervisors). Considering levels of analysis could potentially change the results!
![Page 38: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/38.jpg)
3838
Aggregation
Implications:• So-called “aggregation bias” – when rbetween is
larger than rtotal
– Only occurs if rbetween happens to be larger than rwithin
)1)(1()( 22yxwithinyxbetweentotal rrr
![Page 39: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/39.jpg)
39
Aggregation Bias
Implications:• Don’t look at rtotal to draw inferences about
rwithin!• Don’t look at rtotal to draw inferences about
rbetween!
• See James (1982) and James, Demaree, & Hater (1980), who applied similar formulae to estimate bias in both and corr.’s between aggregated situational (OC) and individual difference variables.
)1)(1()( 22yxwithinyxbetweentotal rrr
![Page 40: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/40.jpg)
40404040
Aggregation BiasSummary:•When we aggregate individual-level measures (e.g., psychological climate) to represent organizational attributes (e.g., organizational climate), then all the theoretical and empirical relationships can change.•Aggregation of the same measures can create a different construct!
![Page 41: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/41.jpg)
414141
Why We Need rWG
• Justifying Aggregation
• “… organizational climate is the overall meaning derived from the aggregation of individual perceptions of a work environment (i.e., the typical or average way people in an organization ascribe meaning to that organization) (James, 1982; Schneider, 1981). Thus, organizational climate can be viewed as the outcome of aggregating individuals’ psychological climates. The important caveat is that these psychological climates are shared in order to make the inference that an organizational climate exists.”
• James et al. (2008, pp. 15-16)
![Page 42: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/42.jpg)
42
Why We Need rWG
Group-Level Consensus Constructs• In measuring group consensus constructs,
agreement and reliability are tools used to justify aggregation of individual-level responses to the group level
• Agreement and reliability help us gauge how well the average across individual responses represents the group.
![Page 43: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/43.jpg)
43
Why We Need rWG
Group-Level Consensus Constructs
Organizational Climate
(average)
Psych. Climate, Person #1
Psych. Climate, Person #2
Psych. Climate, Person #3
![Page 44: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/44.jpg)
44
Why We Need rWG
Overview• Aggregation/Composition Models
– Chan (1998)– Kozlowski & Klein (2000)
• Agreement– rWG family of indices
• Reliability– ICC(1)– ICC(2)
See Bliese, 2000
![Page 45: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/45.jpg)
45
Why We Need rWG
• Aggregation/Composition Models– Chan (1998)– Kozlowski & Klein (2000)
• Both typologies include consensus models– Use the mean of individual responses to
represent the group-level construct– Assume isomorphism (James, 1982)
– Require high within-group agreement
![Page 46: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/46.jpg)
46
Why We Need rWG
• Within-Group Agreement – degree to which ratings from individuals are interchangeable– Agreement-based tests reflect degree to which
raters provide essentially the same rating
– Three dominant indices designed to assess within-group agreement:
• James et al.’s (1984) rWG(J)
• Lindell et al.’s (1999)
• Burke, Finkelstein, & Dusig’s (1999) AD index
*)( JWGr
![Page 47: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/47.jpg)
4747474747474747
George & James (1993)• “The key statistical test of the appropriateness of
aggregation to the group level of analysis is that there is within-group agreement on the variable in question. If there is agreement within groups on the theorized group-level variable, then the aggregate may be used in subsequent analyses.”
• … agreement within a group is not conditional on between-groups differences. For example, in a scenario that Yammarino and Markham portray, in which all members in each group have the same moderately high score, both agreement and aggregation may be justified provided that aggregation to the group level was theoretically based. However, there would be no group effect inasmuch as the group means do not vary under these conditions.”
• (p. 799)
![Page 48: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/48.jpg)
48
Why We Need rWG
• Within-Group Agreement – For single items:
– = observed variance of single item
– = theoretical null variance (represents “zero agreement”)
– rWG = “1 - observed variance over expected variance”
)(1 22ExWG j
sr 2
jxs2E
![Page 49: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/49.jpg)
4949494949
Why We Need rWG
Summary:•Under consensus composition models (with isomorphism across levels), within-group agreement is needed to justify aggregation.
•Within-group agreement is even more essential than ICC(1) and ICC(2), both of which depend upon between-group variance.
•Within-group agreement = shared psychological meaning!
•rWG is the key to measuring group-level psychological properties!
![Page 50: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/50.jpg)
505050
rWG(J) for Multi-Item Scales
rWG(J) is NOT the same as rWG!
•rWG = for single items
•rWG(J) = for multiple-item climate scale
)(1 22ExWG j
sr
)()](1[
)](1[2222
22
)(
ExEx
Ex
JWG
jj
j
ssJ
sJr
![Page 51: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/51.jpg)
51
rWG(J) for Multi-Item Scales
• Within-Group Agreement (James et al., 1984)– For multiple
items:
– J = number of items
– = mean of observed item-level variances
– = theoretical null variance (represents “zero agreement”)
† Can be derived without Spearman-Brown (LeBreton et al., 2005)
)()](1[
)](1[2222
22
)(
ExEx
Ex
JWG
jj
j
ssJ
sJr
2E
2
jxs
![Page 52: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/52.jpg)
52
rWG(J) for Multi-Item Scales
• Three Issues with James et al.’s (1984) rWG(J) :
1)J = number of items
(is rWG(J) an index of agreement, reliability, or both?)
2) = mean of observed item-level variances
3) = theoretical null variance (represents “zero agreement”)
(addressed by LeBreton & Senter, 2008)
2E
2
jxs
![Page 53: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/53.jpg)
535353535353535353
James et al. (1993)• Describing whether rWG(J) is an index of agreement vs.
reliability:• “Kozlowski and Hattrup are also correct in stating that our
intention was to suggest a measure of agreement, and not consistency [reliability], and that rWG is an estimator of agreement. However, what cannot be done, at least not the way things are presently set up, is to follow Kozlowski and Hattrup's recommendation to sever all ties between interrater reliability and rWG and to treat rWG as strictly a measure of agreement with, in effect, no ties to classic measurement theory. It is not possible to follow this recommendation because rWG is currently derived in terms of classic measurement theory as an interchangeability (agreement) index of interrater reliability.”
• (p. 306)
![Page 54: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/54.jpg)
5454
rWG(J) for Multi-Item Scales
• Issues with James et al.’s (1984) rWG(J) :
– J = number of items
• What happens to rWG(J) as number of items (J) increases?
)()](1[
)](1[2222
22
)(
EXEX
EX
JWG
jj
j
ssJ
sJr
![Page 55: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/55.jpg)
555555
rWG(J) for Multi-Item Scales
• Issues with James et al.’s (1984) rWG(J) :
– J = number of items
• What happens to rWG(J) as number of items (J) increases?
)()](1[
)](1[2222
22
)(
EXEX
EX
JWG
jj
j
ssJ
sJr
![Page 56: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/56.jpg)
5656
rWG(J) for Multi-Item Scales
• Issues with James et al.’s (1984) rWG(J) :
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 3 7 11 20
Number of Items (J)
Jam
es e
t al
.'s r
WG
(J)
mean_itemvar = 0.2
mean_itemvar = 0.6
mean_itemvar = 1
mean_itemvar = 1.4
mean_itemvar = 1.8
![Page 57: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/57.jpg)
5757
rWG(J) for Multi-Item Scales
• Issues with James et al.’s (1984) rWG(J) :
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 3 7 11 20
Number of Items (J)
Jam
es e
t al
.'s r
WG
(J)
mean_itemvar = 0.2
mean_itemvar = 0.6
mean_itemvar = 1
mean_itemvar = 1.4
mean_itemvar = 1.8
2
jxs
J
rWG(J) = .7
![Page 58: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/58.jpg)
5858
rWG(J) for Multi-Item Scales
• Issues with James et al.’s (1984) rWG(J) :
– To get a large rWG(J) (James et al., 1984), simply add more items to your scale!!
– Even under near-maximal within-group variance,
[ = 1.8] rWG(J) = .7 when the scale has J = 20 items!2
jxs
![Page 59: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/59.jpg)
59
rWG(J) for Multi-Item Scales
• Issues with James et al.’s (1984) rWG(J) :
– = mean of observed item-level variances
• What is it?
• First calculate the within-group variance of each item,
• Then average these variances across items,
2
jxs
2
jxs2
jxs
![Page 60: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/60.jpg)
60
rWG(J) for Multi-Item Scales
• = mean of observed item-level variances
Compare vs. (scale score variance):
Scale score variance => • First calculate mean across items (i.e., scale score), • Then take the within-group variance of scale score,
• is almost always larger than
2
jxs2
jxs2xs
)]1([22 JJss xx j
2xs
x
2
jxs2xs
2xs
![Page 61: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/61.jpg)
61
rWG(J) for Multi-Item Scales
• Why is almost always larger than scale score variance ?
2
jxs2xs
Psych. Climate
PC Item 1
PC Item 2
PC Item 3
PC Item 4
True Score Variance
Item Unique Variance
![Page 62: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/62.jpg)
62
rWG(J) for Multi-Item Scales
• Why is almost always larger than scale score variance ?
2
jxs
2
jxs
2xs
2xs
Psych. Climate
PC Item 1
PC Item 2
PC Item 3
PC Item 4
True Score Variance
Item Unique Variance
![Page 63: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/63.jpg)
63
rWG(J) for Multi-Item Scales
• = mean of observed item-level variances
Compare vs. (scale score variance):
, Scale score variance => zooms in on true, construct-level variance within-groupsvs.
, Mean of observed item-level variances => includes true construct-level variance + item-specific variance
2
jxs2
jxs2xs
)]1([22 JJss xx j
2xs
2
jxs
![Page 64: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/64.jpg)
64
rWG(J) for Multi-Item Scales
• Issues with James et al.’s (1984) rWG(J) :
– = mean of observed item-level variances
• It would be much clearer to just base within-group agreement on the within-group variance in scale scores, rather than on the average of item-level within-group variances, .
2
jxs
2
jxs2
jxs
![Page 65: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/65.jpg)
65
rWG(J) for Multi-Item Scales
• Issues with James et al.’s (1984) rWG(J) :
– = theoretical null variance (represents “zero agreement”)
– E.g., Uniform null distribution– A = number of response options (e.g., A = 5 for a 5-
point Likert scale);
2E
12)1( 22 AEU
212)15( 22 EU
![Page 66: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/66.jpg)
66
rWG(J) for Multi-Item Scales
• Issues with James et al.’s (1984) rWG(J) : = theoretical null variance
– Can alternatively use a non-uniform expected null variance for rWG(J) (see James et al., 1984; LeBreton & Senter, 2008)
• Normal null dist.
• Skewed null dist.
• Maximum null dist. (Brown & Hauenstein, 2005)
2E
![Page 67: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/67.jpg)
67
rWG(J) for Multi-Item Scales
• Issues with James et al.’s (1984) rWG(J) : = theoretical null variance
– Can alternatively use an Average Deviation index (AD; average absolute value deviation from mean or median; Burke et al., 1999).
• Less vulnerable to outliers
• Still compared against arbitrary cutoff, AD < A/6
• Still includes item-specific variance (like )
2E
2
jxs
![Page 68: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/68.jpg)
6868686868
rWG(J) for Multi-Item Scales
Summary:•Whereas rWG is a great index of standardized within-group agreement,
rWG(J) reflects 3 sources of variance: a) within-group variance in psych. climate/latent
construct true scores (“shared meaning”), plusb) item-specific variance (in ), and c) number of items (J).
•It would be better to use an agreement index that homes in on (a) within-group variance in psych. climate/latent construct true scores (“shared psychological meaning”).
2
jxs
![Page 69: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/69.jpg)
6969
Within-Group Agreement
• So what is the alternative?
![Page 70: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/70.jpg)
70
Within-Group Agreement
• What if we still want to assess within-group agreement (“shared psychological meaning”) with a multi-item climate scale?
• First, conceptualize the degree of “shared psychological meaning” at the latent theoretical level (James, 1982; James et al., 1988), but use a format similar to rWG:
2
2)lim.(1
E
atecpsychWG
![Page 71: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/71.jpg)
7171
Within-Group Agreement
• WG does not increase as you add items to the climate scale (i.e., it is a pure parameter of within-group agreement, not reliability)
2
2)lim.(1
E
atecpsychWG
![Page 72: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/72.jpg)
727272
Within-Group Agreement
• How well does each of the following within-group agreement indices estimate WG? (“shared psychological meaning”)
1)James et al. (1984)
2)Lindell et al. (1999)
3)Simple index:
)()](1[
)](1[2222
22
)(
EXEX
EX
JWG
jj
j
ssJ
sJr
)(1 22*)( EXJWG j
sr
)(1 22)( ExWG sr
![Page 73: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/73.jpg)
737373
Within-Group Agreement• Comparison of rWG(J), rWG(J)*, and rWG()
Newman & Sin, 2008
J =5 items, WG = .90
![Page 74: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/74.jpg)
74747474
Within-Group Agreement
• Conclusions:
1)All within-group agreement indices are very strongly correlated.
2) rWG(J) can notably overestimate within group agreement, especially when rWG(J) > .7.
3)rWG() seems to offer a closer estimate of within group agreement (slight underestimate)
4)One could also directly estimate WG .
![Page 75: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/75.jpg)
75757575
Within-Group Agreement
• How well does each of the following within-group agreement indices estimate WG? (“shared psychological meaning”)
1)When WG = .60:
rWG(J)= .75; rWG(J)*= .38, rWG() = .56
2)When WG = .65:
rWG(J)= .81; rWG(J)*= .46, rWG() = .61
3)When WG = .70:
rWG(J)= .85; rWG(J)*= .53, rWG() = .67
J =5 items, WG = .90
![Page 76: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/76.jpg)
767676
OverviewGroup-Level Psychological Properties?
1.Psychological Climate Group-Level vs. Individual-Level Constructs
2.Aggregation Bias
3.Why we need rWG (Within-group agreement) Justifying Aggregation
4.rWG(J) for multi-item scales Agreement vs. Reliability
![Page 77: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/77.jpg)
777777
OverviewGroup-Level Psychological Properties?
1.Psychological Climate Group-Level vs. Individual-Level Constructs
2.Aggregation Bias
3.Why we need rWG (Within-group agreement) Justifying Aggregation
4.rWG(J) for multi-item scales Agreement vs. Reliability
![Page 78: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/78.jpg)
787878
OverviewGroup-Level Psychological Properties?
1.Psychological Climate Group-Level vs. Individual-Level Constructs
2.Aggregation Bias
3.Why we need rWG (Within-group agreement) Justifying Aggregation
4.rWG(J) for multi-item scales Agreement vs. Reliability
![Page 79: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/79.jpg)
797979
OverviewGroup-Level Psychological Properties?
1.Psychological Climate Group-Level vs. Individual-Level Constructs
2.Aggregation Bias
3.Why we need rWG (Within-group agreement) Justifying Aggregation
4.rWG(J) for multi-item scales Agreement vs. Reliability
![Page 80: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/80.jpg)
808080
OverviewGroup-Level Psychological Properties?
1.Psychological Climate Group-Level vs. Individual-Level Constructs
2.Aggregation Bias
3.Why we need rWG (Within-group agreement) Justifying Aggregation
4.rWG(J) for multi-item scales Agreement vs. Reliability
![Page 81: 1 Measuring Group-Level Psychological Properties (A Tribute to Larry James) Daniel A. Newman University of Illinois Daniel A. Newman, Ph.D](https://reader030.vdocuments.mx/reader030/viewer/2022012922/56649dff5503460f94ae834d/html5/thumbnails/81.jpg)
8181
Thank You Larry!OC
(average)
PC Person #1
PC Person #2
PC Person #3
)(1 22ExWG j
sr