ray poynter advances in quant - 2011
TRANSCRIPT
![Page 1: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/1.jpg)
A Presenta*on from The NewMR “Advances in Quan*ta*ve Research” Event
19 September, 2012
Event sponsored by Affinnova All copyright owned by The Future Place and the presenters of the material
For more informa=on about Affinnova visit www.affinnova.com For more informa=on about NewMR events visit newmr.org
Scales in online surveys – An updated set of best prac=ces Ray Poynter, Vision Cri0cal University
![Page 2: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/2.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Agree/Disagree scales in online surveys LeK, right, top, and middle thoughts
Ray Poynter Director Vision Cri=cal University
@ Vision Cri=cal
![Page 3: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/3.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Agree/Disagree Scales Sometimes called Likert Scales
– More strictly Likert Items
Typical format – Agree strongly – Agree – Neither agree nor disagree – Disagree – Disagree strongly
Many variants exist
![Page 4: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/4.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Issues
Acquiescence bias
Left versus right
Top versus bottom
Mid-points
![Page 5: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/5.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Acquiescence Bias Saris, Krosnick, and Shaeffer http://goo.gl/LvAwS
Too many people agree!
Traditional solution – Include each factor several times, with slightly
different wordings – Phrasing half positive, half negative
Not practical in modern market research – Ask positively and only once
![Page 6: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/6.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Item Order Response Bias Many sources (e.g Krosnick, Chan, Friedman etc)
Items that appear first are more likely to be accepted
A Agree Neither Agree nor Disagree Disagree B Disagree Neither Agree nor Disagree Agree
In A there will be more Agrees In B there will be more Disagrees
![Page 7: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/7.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Left to Right To what extent do you agree or disagree with the statement Diet Coke, Orange juice, Milk and Red wine are healthy drinks?
A
B
Cells approx. 500, Vision Cri=cal Springboard UK Omni, 2012
![Page 8: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/8.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Left to Right Diet Coke Healthy
5%
10%
23%
35%
27%
2%
11%
24%
33% 31%
Agree strongly Agree Neither Nor Disagree Disagree Strongly
Diet Coke A Diet Coke B +/-‐ 3% = sig dif
![Page 9: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/9.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Left Versus Right
% Agree Strongly Agree on Left Agree on Right Diet Coke 5% 2% Orange Juice 21% 16% Milk 38% 30% Red Wine 6% 4%
% Disagree Strongly Disagree on
Right Disagree on
Left Diet Coke 27% 31% Orange Juice 1% 1% Milk 1% 1% Red Wine 5% 9%
People (in this context, in the UK) are more likely to pick the le` side of the scale
![Page 10: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/10.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Real but not big!
Healthy? %
Agree Mean
(5 points) Milk A 89% 4.2 Milk B 83% 4.1 Orange Juice A 76% 3.9 Orange Juice B 72% 3.8 Red Wine A 36% 3.1 Red Wine B 33% 3.0 Diet Coke A 15% 2.3 Diet Coke B 12% 2.2
Agree = Agree + Strongly Agree
![Page 11: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/11.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Observations so far 1. Acquiescence bias leads to higher
agreement
2. Putting agree on left increase the bias
3. Putting agree on right reduces the bias
4. The effect is not very large
5. So, DON’T MIX AND MATCH
![Page 12: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/12.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Top to Bottom A B
![Page 13: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/13.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Top to Bottom Water Healthy
83%
16%
1% 0% 0%
66%
29%
2% 2% 2%
Agree strongly Agree Neither Nor Disagree Disagree Strongly
Water A Water B +/-‐ 3% = sig dif
![Page 14: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/14.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Top Versus Bottom
% Agree Strongly Agree at Top Agree at Bottom Water 83% 66% White Wine 3% 1% Apple Juice 19% 13% Coke 2% 1%
% Disagree Strongly Disagree at
Bottom Disagree at
Top Water 0% 2% White Wine 5% 11% Apple Juice 0% 2% Coke 39% 46%
People (in this context, in the UK) are more likely to pick the top items
![Page 15: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/15.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Don’t Mix and Match
Horizontal -‐ standard Ver*cal -‐ visual % Agree + Agree Strongly
Agree Le`
Agree Right
Agree Top
Agree Bofom
Water 95% 94% 98% 95% White Wine 19% 14% 26% 18% Apple Juice 66% 66% 77% 70% Coke 5% 6% 6% 5%
![Page 16: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/16.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Response Order Thoughts
There are order effects
Putting Agree on the left/top compounds acquiescence
AOTBE put Disagree left/top
Don’t mix and match
![Page 17: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/17.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Mid-points
![Page 18: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/18.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Adding a Don’t Know
DKs: Water 0%, Red Wine 2%, Milk 0%, Coffee 0%, Diet Coke 0%
![Page 19: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/19.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Moving the Neither Nor
![Page 20: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/20.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Moving the Neither Nor
0%
10%
20%
30%
40%
50%
60%
70%
80%
Disagree Strongly
Disagree Neither agree nor ..
Agree Agree Strongly
Water -‐ Mid Water -‐ End
![Page 21: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/21.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Moving the Neither Nor
0% 5%
10% 15% 20% 25% 30% 35% 40% 45%
Disagree Strongly
Disagree Neither agree nor ..
Agree Agree Strongly
Red Wine -‐ Mid Red Wine -‐ End
![Page 22: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/22.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Moving the Neither Nor
0%
10%
20%
30%
40%
50%
60%
Disagree Strongly
Disagree Neither agree nor ..
Agree Agree Strongly
Milk -‐ Mid Milk -‐ End
![Page 23: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/23.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Moving the Neither Nor
0%
10%
20%
30%
40%
50%
60%
Disagree Strongly
Disagree Neither agree nor ..
Agree Agree Strongly
Coffee -‐ Mid Coffee -‐ End
![Page 24: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/24.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Moving the Neither Nor
0% 5%
10% 15% 20% 25% 30% 35% 40% 45% 50%
Disagree Strongly
Disagree Neither agree nor ..
Agree Agree Strongly
Diet Coke -‐ Mid Diet Coke -‐ End
![Page 25: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/25.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Neither Nor
In this sort of scale – Where the ends are opposites – And where people who do not agree or
disagree are likely to be in the middle – Neither agree nor disagree works well as a
mid-point in a five point scale Even if it does not always make semantic sense!
![Page 26: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/26.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Concluding Thoughts 1. The way we ask questions shapes the
answers
2. We can’t remove bias – Recognise it – Hold it constant
3. Most standard formats work reasonably well
4. But don’t mix and match!
![Page 27: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/27.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Thank you Ray Poynter
Director Vision Cri=cal University @ Vision Cri=cal
![Page 28: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/28.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Q & A
Sue York NewMR
Ray Poynter Vision Critical University
![Page 29: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/29.jpg)
Speaker: Ray Poynter, Vision Critical University, UK NewMR Advances in Quantitative Research Event, 19 September 2012, Session 1
Ray Poynter Vision Cri*cal University @Vision Cri*cal www.visioncri*caluniversity.com ray.poynter@visioncri*cal.com @RayPoynter
www.linkedin.com/in/raypoynter
![Page 30: Ray poynter advances in quant - 2011](https://reader030.vdocuments.mx/reader030/viewer/2022020116/55cadeeabb61eb7a148b461f/html5/thumbnails/30.jpg)
A Presenta*on from The NewMR “Advances in Quan*ta*ve Research” Event
19 September, 2012
Event sponsored by Affinnova All copyright owned by The Future Place and the presenters of the material
For more informa=on about Affinnova visit www.affinnova.com For more informa=on about NewMR events visit newmr.org
Scales in online surveys – An updated set of best prac=ces Ray Poynter, Vision Cri0cal University