scale perception bias in likert scale questions: improved validity through anchoring vignettes?
DESCRIPTION
Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?. Klaus Gebel Centre for Chronic Disease Prevention James Cook University. German tourists in Spain. Political efficacy in China and Mexico. - PowerPoint PPT PresentationTRANSCRIPT
Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?
Klaus Gebel
Centre for Chronic Disease Prevention
James Cook University
German tourists in SpainGerman tourists in Spain
Political efficacy in China and MexicoPolitical efficacy in China and Mexico
Health inequalities between Indigenous and Health inequalities between Indigenous and non-Indigenous Australiansnon-Indigenous Australians
Anchoring vignettesAnchoring vignettesExample: Self-rated healthExample: Self-rated health
ExcellentExcellent Very goodVery good GoodGood FairFair PoorPoor
Most common measure of health Often negative association with actual health
SF-36: In general, would you say your health is:
Anchoring vignettesDifferential Item Functioning (DIF)
SelfSelf11
Low Low
HighHigh
Jo1
Pat1
Chris1
SelfSelf22
Jo2
Pat2
Chris2
Low
High
Chris2
Jo2
Pat2
SelfSelf22
Recoding depending on self-rating and rating of vignettes
Self & vignettes Recoding
x>V1>V2>V3 7x=V1>V2>V3 6V1>x>V2>V3 5V1>x=V2>V3 4V1>V2>x>V3 3V1>V2>x=V3 2V1>V2>V3>x 1x>V1>V2=V3 7x>V1=V2=V3 7
Next steps
Conversion from 7-point scale back to 5-point scale
Paired t-test to compare DIF-adjusted vs unadjusted scores
Wald’s test to compare associations of DIF-adjusted vs unadjusted scores with continuous or binary outcome measures
Design of vignettes
Ideally equally spaced through distribution of self-assessments
Avoid extremes Gender specific Ideally 2-3 vignettes Design vignettes in focus group Test them with sub sample
Example: Safety from traffic and cycling / walking
Safety and accessibility are the two most important environmental factors affecting activity participation across the lifespan.
DiPietro 2012, Human Kinetics
“”
Background
Background
Safety from traffic
Speed of traffic
Volume of traffic
Separation from traffic
Results
Very unsafe
A little unsafe
Neither safe nor unsafe
A little safe
Very safe
How safe do you think it is to ride a bicycle in your local area?
1 2 3 4 5
Predictive validity of perceived safety from traffic
Methods
Baseline data collected 2013Cycling and walking measured with travel diary
Outcome measures
• Cycle to work/study y/n (Probit)
• Frequency of cycling (ordered Probit)
• Mins/week of cycling and walking (Poisson)
Results
n=871 58% women 18-55 (37±11.1 years) 14% use bicycle as main way travel mode to
work/study 29% cycle at least 1-2 days/week
Results
Perceived safety from traffic
Results
Ratings of vignettes
0
100
200
300
400
500
600
Very unsafe A littleunsafe
Neithersafe norunsafe
A little safe Very safe0
50
100
150
200
250
300
350
Very unsafe A little unsafe Neither safenor unsafe
A little safe Very safe0
100
200
300
400
500
600
Very unsafe A little unsafe Neither safenor unsafe
A little safe Very safe
Results
DIF adjusted perceived safety from traffic
Results
Unadjusted Adjusted for DIF
Wald’s test p-value
Cycle to work/study y/n 0.63 0.79 0.03
Frequency of cycling 0.71 0.78 0.06
Mins/week of cycling 0.53 0.78 >0.01
Mins/week of walking 0.48 0.62 0.01
Predictive validity of perceived safety from traffic
Conclusion
Significant scale perception bias Anchoring vignettes predictive validity Anchoring vignettes might be a powerful tool
to improve validity of Likert-scale items
Opportunistic evaluation of new bicycle paths in Cairns
Bike path evaluation in CairnsBike path evaluation in Cairns
Thank you for your attention!Thank you for your attention!
Questions?Questions?