How to evaluate the cross-cultural equivalence of single items

Melanie Revilla, Willem SarisRECSM, UPF

Zurich – 15/16 July

Cross-cultural equivalence• Usually

– Discussed in the frame of cross-national research– Idea: in different countries people can express themselves in

different ways– Different cultures can also be defined on other criteria (e.g. language)– Procedure: can be applied in similar way to all kinds of different

groups

• “Equivalence” measurement equivalence – 2 persons with the same opinion will give the same answer (whatever

their group)

• Important– because observed differences may result from non equivalent

measures and not be real differences

• If measurement equivalence does not hold– cannot make comparison across groups!!

Important distinction (Northrop, 1947)

• Concept by Intuition (CI)

– Simple concepts that can be measured directly– Single item– Ex: trust in the parliament

• Concept by Postulation (CP)

– Complex concepts that cannot be measured directly– Also called “construct”– Need several CI to measure them– Ex: political trust: trust in parliament + legal system + police

…

• Classic procedure to test for equivalence for CP but not for CI

start with a reminder of the procedure for CP

When we have multiple indicators

CP / complex concepts

Basic Confirmatory Factor Analysis model

CP1

Y1

Y2

Y3

λ11

λ21

λ31

e1

e2

e3

τ1

τ3

τ2

Yi = τi + λij CP1 + ei i = 1,2,3

Political trust

Answer Trust in the parliament

Answer Trust in the legal system

Answer Trust in the police

intercepts slopes error terms

≈ regression equation [

Dependent variable

Independent variable

Multiple Group CFA approach

Group 1 Group 2

• Multiple group: – possible to test for equality of the parameters in the

different groups – constraints across groups

• Can be extended to more groups

Different levels of invariance (Meredith, 1993)

• Configural– Same model holds in all

groups

• Metric– Configural + Slopes (λij) the

same in all groups– Sufficient for comparison of

relationships

• Scalar– Metric + Intercepts (τi) the

same in all groups– Sufficient for comparison of

means

• More: error terms, etc…

Group 1

Group 2

In practice

• Analyses can be done with standard SEM softwares– LISREL/Mplus– based on covariance matrices & means– recommended sample size: >200 in each

group– 3-step procedure: configural, metric, scalar– syntax quite easy to get estimates

• More tricky but crucial step: testing

Testing the model• Assessing global fit

– Chi2 test / Fit indices: RMSEA (<.05), CFI (>.9), etc…– Limits: Depends on sample size / Sensitive to deviations from

normality

• Assessing local fit

– Saris & Satorra should test at the parameter level + take into account type II errors (H0 not rejected despite being false)

– JRule software (van der Veld, Saris, Satorra) + Jrule for Mplus (Oberski)• See next presentation!

• Always check if estimates are really different

– Difference may be statistically significant but not substantially meaningful

• Partial invariance

– What if some indicators are equivalent but not all?– Consistent estimates of the means of the latent variables if at least

2 indicators are scalar invariant (Byrne, Shavelson, Muthén, 1989)

If we have single indicator

CI / simple concepts

Single items

Y1

e1

τ1CI1 λ11

• Testing equivalence single items

• Testing equivalence for CI

Yi = τi + λij CIi + ei

Y1

e1

τ1CI1 λ11

Group 1 Group 2

Single multiple indicators?

• Problem: model just presented not identified

• “Single indicator” = single trait in fact

• But possible to use multiple methods

• So for CI:– Only one trait, but we can always have more

than one method– Several indicators = same trait asked using

different methods

Can apply again MGCFA

Group1 Group 2

• Similar at the previous model (for CP) but now different methods instead of different traits measuring a same concept

Trust in theparliament

11 points

6 points

4 points

Same procedure

• Different levels of invariance as for CP

– Configural – Metric– Scalar– etc

• Same procedure to get the estimates and test the model

– Multiple group analyses– Test of the model: global / local fit– Partial equivalence

Problem for the CI

• Fix the scale?

– As before, necessary to fix the scale of the LV– Usually, fix the first loading to 1– Can be done here too– Other loadings are relative to the first one– But need to be done in all groups– If there are differences for the method whose

loading is fixed to 1 across groups, may be problematic

– Should try to use methods that have been shown to be the most similar across groups: e.g. fixed reference points

General model

CP1

Y31

Y22

Y13

c1

c2

c3

e11

e12

e33

τ11

τ33

τ12

CI1

CI2

CI3

v21

v22

v23

u1

u2

u3

α1

α2

α3

• Even when working on CP: better to use different methods

Y21

Y12

Y23

Y33

Y11

Y32

e21

τ21

e31

τ31

e23

τ23

e13

τ13

e22

τ22

e32

τ32

1

v31

v32

v33

1

1

Final remarks / CCL

Equivalence single items

• Need to repeat the same item with different methods

– 3 or more repetitions

– Multi Methods (MM)?• Same persons get the question several times using different

methods • Limit: 20 minutes at least to avoid memory effects (Van Meurs &

Saris, 1990)

– Mix with Split-Ballot (SB) design? • Random assignment of respondents to different versions of the

questionnaire• “SB-MM” (CI) or SB-MTMM (CP)?

Conclusion

• Measurement equivalence can be assessed both for CP and CI using Multiple Group Confirmatory Factor Analysis

– For CP, process already well-known and used a lot– For CI, possible do similar analyses

But necessary to repeat questions!! specific data

• So testing equivalence of simple item can be done using a (SB)-(MT)MM approach– Similar to what exists in the ESS for CP: main +

supplementary questionnaires (different versions)– With extension for concepts by intuition

In summary

• To test single item equivalence

– Use multiple methods

– Do everything as for multiple items equivalence

Thank you for your attention!

Questions?