BAD BOY MATRIX QUESTION Whatcha gonna do when they come for you?
Florian Tress
ODC Services GmbH, Germany
GENERAL ONLINE RESEARCH 2012
5 – 7 March 2012 at the DHBW Mannheim
2 THE VENN DIAGRAM OF FIELDWORK
PEOPLE RESEARCHERS
Fun & entertainment
Incentives
Disport
Curiosity
Express an opinion
Solve problems
Collect (a lot of) valid data
Gain a lot of knowledge
Implement special
(and possibly boring)
methods
3 BAD BOY MATRIX QUESTION
PEOPLE
Monotonous, boring
Overwhelming variety of options: “decision paralysis”
Nondifferentiation, satisficing behavior
Another question format would be more appropriate: Inferior data
RESEARCHERS
Standardized data format, comparability of responses (with common orientation)
Maximize amount of data, minimize length of interview
Facilitate statistical procedures, e.g. factorial analysis, indices, etc.
Availability of validated test instruments , e.g. Big Five
4 STANDARD ALTERNATIVES
MULTIPLE CHOICE
Matrix with a two point scale
Additional option “None of these” (mandatory question)
RANKING
Bring statements in an order
Perfectly differentiated data
5 SPECIAL ALTERNATIVE: THE CAROUSEL
Only one statement presented at once
New statements slide in from the left
Response options remain in the same place
6 SPECIAL ALTERNATIVE: DRAG ME
All statements presented at once
Center stands for respondent
Measures distance from center
Arrangement of statements unimportant
7 COMPARISON OF THESE ALTERNATIVES
Monadic Questionnaire : Random assignment to one of these alternatives
n = 1080; 216 interviews per alternative; good spread over age and education
Indicators: Comparability, Trustworthiness, Data Quality, Satisfaction, Technical Requirements
8 COMPARABILITY: BRAND LIKEABILITY
39
39
43
47
52
53
54
62
66
73
McDonalds
Siemens
BMW
Coca Cola
Nutella
Volkswagen
IKEA
Nivea
Amazon
Multiple Choice
3,2
3,7
3,8
3,9
4,2
4,2
4,6
5,0
5,8
6,6
McDonalds
Siemens
IKEA
Nutella
Coca Cola
BMW
Volkswagen
Nivea
Amazon
Ranking
3,1
3,5
3,5
3,5
3,5
3,6
3,7
3,9
3,9
4,0
McDonalds
IKEA
BMW
Siemens
Coca Cola
Nutella
Volkswagen
Nivea
Amazon
Matrix
44
48
49
51
52
52
57
61
66
68
McDonalds
BMW
Nutella
Volkswagen
Siemens
Ikea
Coca Cola
Nivea
Amazon
Drag Me
3,2
3,4
3,4
3,5
3,5
3,5
3,7
3,9
4,1
4,1
McDonalds
Nutella
Coca Cola
BMW
Siemens
Ikea
Volkswagen
Nivea
Amazon
Carousel
3 F
acto
rs (
Cum
.%:
49)
| C
ronbach„s
α:
n.a
. 3 F
acto
rs (
Cum
.%:
56)
| C
ronbach„s
α:
0,7
7.
3 F
acto
rs (
Cum
.%:
58)
| C
ronbach„s
α:
0,7
8.
3 F
acto
rs (
Cum
.%:
57)
| C
ronbach„s
α:
0,7
3.
3 Factors (Cum.%: 60) | Cronbach„s α: 0,80.
9 COMPARABILITY: ATTITUDES
11
14
18
24
28
28
30
38
44
62
I
H
J
G
B
D
F
E
C
A
Multiple Choice
3,0
3,4
3,6
3,8
4,0
4,6
4,9
4,9
6,4
6,4
G
H
I
J
B
F
E
D
A
C
Ranking
2,8
2,8
3,0
3,1
3,2
3,2
3,5
3,5
3,6
3,8
J
I
H
G
F
E
D
C
B
A
Matrix
32
39
41
44
49
50
52
55
63
65
H
I
J
B
D
G
F
E
C
A
Drag Me
2,9
3,0
3,1
3,1
3,4
3,4
3,5
3,5
3,7
3,9
I
J
G
H
F
E
D
B
C
A
Carousel
(A) Wenn ich gute Erfahrungen mit einer Marke mache, empfehle ich sie aktiv weiter. (B) Werbung sollte mich stärker über Marken informieren, die ich noch nicht kenne. (C) Ich
habe feste Marken, die ich bevorzugt einkaufe. (D) Neuartige und innovative Produkte passen gut zu meinem Lebensstil. (E) Ich probiere häufig neue Marken aus, die ich noch
nicht kenne. (F) Ich bevorzuge Marken, die auf eine lange Tradition zurückblicken. (G) Ich bin bereit, für Markenprodukte mehr Geld auszugeben. (H) Werbung sollte mich stärker
über Marken informieren, die ich bereits gut kenne. (I) Je bekannter eine Marke ist, desto leichter kann man ihr vertrauen. (J) Die Bekanntheit einer Marke sagt etwas ihre Qualität aus.
3 F
acto
rs (
Cum
.%:
51)
| C
ronbach„s
α:
n.a
. 3 F
acto
rs (
Cum
.%:
50)
| C
ronbach„s
α:
0,5
6.
3 F
acto
rs (
Cum
.%:
61)
| C
ronbach„s
α:
0,8
0.
3 F
acto
rs (
Cum
.%:
63)
| C
ronbach„s
α:
0,8
2.
3 Factors (Cum.%: 63) | Cronbach„s α: 0,82.
10 RESULT: COMPARABILITY
WINNER
Results correspond (roughly) for the most / least likeable brands (agreeable statements)
But: Drag Me seems to measure something different / to be biased by third variables
LOSER
11 TRUSTWORTHINESS: BRAND LIKEABILITY
MULTIPLE CHOICE
Ø Words pos.: 5,5
Ø Words neg.: 5,4
Non-Response: 3%
MATRIX
Ø Words pos.: 6,8
Ø Words neg.: 7,3
Non-Response: 2%
RANKING
Ø Words pos.: 6,3
Ø Words neg.: 6,7
Non-Response: 3%
CAROUSEL
Ø Words pos.: 7,7
Ø Words neg.: 7,0
Non-Response: 3%
DRAG ME
Ø Words pos.: 7,2
Ø Words neg.: 7,2
Non-Response: 2%
Follow-Up-Exploration: Why do you think, this is the most / least likeable brand?
12 TRUSTWORTHINESS: ATTITUDES
MULTIPLE CHOICE
Ø Words pos.: 7,4
Ø Words neg.: 8,6
Non-Response: 11%
MATRIX
Ø Words pos.: 7,9
Ø Words neg.: 9,6
Non-Response: 9%
RANKING
Ø Words pos.: 8,3
Ø Words neg.: 8,1
Non-Response: 7%
CAROUSEL
Ø Words pos.: 9,3
Ø Words neg.: 10,0
Non-Response: 8%
DRAG ME
Ø Words pos.: 8,4
Ø Words neg.: 8,6
Non-Response: 7%
Follow-Up-Exploration: Why did you agree / disagree with this statement?
13
WINNER
RESULT: TRUSTWORTHINESS
LOSER
nonspecific / general
„because I like it“
specific / rich in detail
„good quality, fair prices“
14 DATA QUALITY: NONDIFFERENTIATION
13%
0%
6%
4%
0%
8%
0%
8%
5%
0%
Multiple Choice Ranking Matrix Carousel Drag Me
Brand Likeability Attitudes
Low sample size!
15 DATA QUALITY: NONDIFFERENTIATION
2% 5%
3%
3%
Brand Likeability Attitudes
2% 2%
2% 2%
Brand Likeability Attitudes
6% 3%
7%
4%
Brand Likeability Attitudes
6%
2%
7%
5%
Brand Likeability Attitudes
3% 5%
2%
3%
Brand Likeability Attitudes
2% 3%
2% 1%
Brand Likeability Attitudes
MULTIPLE CHOICE MATRIX CAROUSEL
Educati
on
Age
younger
40
old
er
low
er
A-L
. h
igher
Low sample size!
16
WINNER
RESULT: DATA QUALITY
LOSER
The data is perfectly differentiated with the Ranking and Drag Me Question.
Among the other three alternatives, the Carousel performs best.
17 SATISFACTION
66
70
71
73
75
28
23
18
20
19
3
6
8
6
3
Multiple Choice
Matrix
DragMe
Ranking
Carousel
Usability
63
65
66
67
71
31
25
23
22
20
4
9
10
11
5
Multiple Choice
Matrix
DragMe
Ranking
Carousel
Layout
60
61
64
66
69
22
21
19
18
15
15
15
11
11
13
Matrix
Multiple Choice
DragMe
Ranking
Carousel
Length
54
54
55
57
64
12
15
16
14
15
14
11
11
11
8
Matrix
Multiple Choice
DragMe
Ranking
Carousel
Topic
53
54
54
57
62
14
16
13
13
14
12
11
12
10
10
Multiple Choice
DragMe
Matrix
Ranking
Carousel
Fun
67
70
70
71
74
17
16
16
15
15
4
5
4
4
2
Multiple Choice
DragMe
Matrix
Ranking
Carousel
Comprehensibility
18 SATISFACTION
45
46
57
58
59
35
34
25
25
24
17
15
15
16
14
Multiple Choice
Matrix
DragMe
Ranking
Carousel
Overall
WINNER LOSER
19 SUMMARY
MULTIPLE
CHOICE RANKING MATRIX CAROUSEL DRAG ME
Comparability
Trustworthiness
Data Quality
Satisfaction
Technical
Requirements none JScript none JScript, Flash JScript, Flash
20 RECOMMENDATIONS
CHECK, IF YOUR STUDY PERMITS THE USAGE OF JSCRIPT AND FLASH!
(in most cases, it will)
SELECT THE QUESTIONTYPES CAREFULLY!
(there might be better alternatives)
LAYOUT AND USABILITY MATTER!
(the longer the interview, the more they matter)
IF YOU HAVE DOUBTS ASK YOUR FIELDWORK PROVIDER!
(they should have enough experience)
