William van der Veld (University of Amsterdam)Willem Saris (ESADE Business School)Barcelona, 2005, European Association for Survey Research

THE NATURE OF MEASUREMENT ERRORIN PANEL DATA

Reliability and Opinion Stability

2

Overview• The RUSSET panel (www.vanderveld.nl)• Reliability estimation in a panel design• Reliability estimation in a cross-section

design• Comparison of Panel & Cross-section• The VAS Model• Conclusion

3

The RUSSET panelPlease, tell me, how satisfied are you with your current life as a whole?

1 2 3 4 5 6 7 8 9 10 Not at all Very Satisfied Satisfied

123456

1994 1995 1998

Aggregate level [means] Individual level (Correlations)

1993 1995 19981993 1.001995 0.34 1.001998 0.30 0.27 1.00

4

Reliability estimation in a panel design

Quasi-simplex model: Assumptions• The random error variance is the identical for the repeated measures;• For each respondent the attitude changes according to a lag-1 quasi-

simplex;• The repeated measures are independent.

T1

y1

λ1

β21 T2

y2

λ2

β32 T3

y3

e

λ3

e e

d2 d3 Interpretation• y: The observed variable• e: the random error in y• T = y + e• When T and y standardized,• β: the stability coefficient• λ: reliability coefficient

5

Reliability estimation in a panel design• Estimation of Reliability & Stability• RUSSET Data: 3 waves, 3 times the question:• How satisfied are you with your current life…?• n=837; Chi-square=0; df=0 [Lisrel, ML estimation]

Completely Standardized Solution

λ1 λ2 λ3 s21 s32 0.59 0.58 0.59 0.99 0.82

1993 1995-6

1998

1993 1.001995-6

0.34 1.00

1998 0.30 0.27 1.00

6

Reliability estimation in a panel design• The observed stability is:

0.34 & 0.27.• After correction for measurement error, the

stability is:0.99 & 0.82.

• The reliability coefficient of a simple question as ‘How satisfied are you with your current life as a whole?’ is quite poor: ±0.6 (reliability=0.36)

• Is there an alternative way to estimate the reliability,so that we can compare both estimates?

7

Reliability estimation in a cross-section design

Parallel-test model: Assumptions• The random error variance is the identical for the

repeated measures;• For each respondent the attitude has not changed

during the interview;• The repeated measures are independent.

Interpretation• y: The observed variable• e: the random error in y• T = y + e• When T and y

standardized,• λ: reliability coefficient

λ1 λ2

x1 x2

T

e e

8

Reliability estimation in a cross-section design• Estimation of Reliability (not Stability)• RUSSET Data: 1 wave, 2 times the question:• How satisfied are you with your current life…?• Same respondents (exactly), same question.• n=837; Chi-square=1; df=1 [Lisrel, ML estimation]

1995-6 995-1371995-6 1.001995-137

0.72 1.00

Completely Standardized Solution

λ1 λ2-6 λ2-137 λ3 - 0.85 0.85 -

9

Reliability estimation in a cross-section design• THAT’S STRANGE!!!!• Same question, same respondents, same moment• Same definition:

• Still estimated reliability PTR(6)=0.85 vs. QSR(6)=0.58• No confirmation because they are different.• So, this result does not help• Is there a way to check which reliability is correct?

10

Comparison of QSR & PTR• Use the reliability-estimates to

correct the observed correlation between different variables for measurement error.

• ρ21 = r21/(λTx1 * λ Tx2)• We have observed the

correlation: r21• We have estimated the reliability

coefficients with QS & PT model.• We are interested in: ρ21• Which ρ21 is more plausible?

That obtained with the QS-estimates or that with the PT-estimates

Ts1

xs1

λTx1

ρ21 Ts

2

xs2

λTx2

e'2 e'

1 r21

Tx21 21Tx21 ρr

11

Comparison of QSR & PTR• Exactly the same data (n=837), Wave 1995(6,7).• Correction of the observed correlation between:

Satlife & Satinc.• Satlife:

How satisfied are you with your current life as a whole?

• Satinc:How satisfied are you with your family’s current financial situation?

• Observed correlation between Satlife & Satinc = 0.51

• The reliability coefficient estimates are:

12

Comparison of QSR & PTRStandardized estimates of reliability coefficients

QSRC PTRCλ1 λ2-6 λ3 λ2-6 λ2-m2

SATLIFE 0.59 0.58 0.59 0.85 0.85SATINC 0.57 0.53 0.63 0.84 0.84Cor(Satlife, Satinc)=

.51 .51

ρ21=

13

Comparison of QSR & PTRStandardized estimates of reliability coefficients

QSRC PTRCλ1 λ2-6 λ3 λ2-6 λ2-m2

SATLIFE 0.59 0.58 0.59 0.85 0.85SATINC 0.57 0.53 0.63 0.84 0.84Cor(Satlife, Satinc)=

.51 .51

ρ21=

.71

14

Comparison of QSR & PTRStandardized estimates of reliability coefficients

QSRC PTRCλ1 λ2-6 λ3 λ2-6 λ2-m2

SATLIFE 0.59 0.58 0.59 0.85 0.85SATINC 0.57 0.53 0.63 0.84 0.84Cor(Satlife, Satinc)=

.51 .51

ρ21=

1.66 .71

15

Comparison of QSR & PTR• It appears that the QSR-estimates are wrong.

What’s wrong with the quasi-simplex reliability?

• The key is provided by the VAS model (Van der Veld & Saris, 2004)

• It can be derived that: QSR=q(vas)*c(vas)• This is proven in the accompanying paper, but

it can be illustrated too.

16

The VAS ModelFor details, visit

Session:Nonattitudes and informed opinions at Thursday 11:30Chair: Peter Neijens

17

The quasi-simplex model

T1

y1

λ1

β21 T2

y2

λ2

β32 T3

y3

e

λ3

e e

d2 d3

18

The VAS ModelFor details, visit

Session:Nonattitudes and informed opinions at Thursday 11:30Chair: Peter Neijens

19

The VAS Model - Estimation of Q, S, & C• RUSSET Data: 3 waves, 6 times the question (n=627).• Satlife:

How satisfied are you with your current life as a whole? • Satinc:

How satisfied are you with your family’s current financial situation?

Completely Standardized Solution χ2(6) q11 q12 q21 q22 q31 q32 c1 c2 c3 s21 s32 SATLIFE 14.19 0.85 0.88 0.81 0.91 0.83 0.84 0.79 0.75 0.76 0.81 0.79 SATINC 34.92 0.80 0.90 0.78 0.94 0.90 0.89 0.73 0.70 0.81 0.81 0.66

20

Comparison of QSR & PTR & VASStandardized estimates of reliability coefficients

QS PT VASλ2-6 λ2-6 q2-6 c2-6 q*c

SATLIFE 0.58 0.85 0.81 0.75 0.60SATINC 0.53 0.84 0.78 0.70 0.54

• QSRC=q*cQSRCsatlife=.81*.75=.60QSRCsatinc=.78*.70=.54

21

Conclusion• The quasi-simplex model has parameters for

– Stability– Reliability

• The reliability in that model is not correct, It is the product of:– The quality of the measurement

instrument– The opinion crystallization

• The stability is correct [see paper].

22

Conclusion• How do you feel about public security these days?• With the QSM all the considerations that are unique

for a specific time are not in the variable S. Hence such considerations are not part of the stability.

• One should be aware of this.• It depends on the object of study, whether this

model is appropriate to estimate the stability!• It is definitely not correct for reliability estimation!