a tale of two surveys: how using real words instead of mumbo jumbo affects survey data quality
TRANSCRIPT
#CRC2014 @LoveStats
A Tale of Two Surveys
Presented by
Annie Pettit, Chief Research Officer
Peanut Labs is a
company
#CRC2014 @LoveStats
It was the best of times, it was the
worst of times, it was the age of
wisdom, it was the age of foolishness,
it was the epoch of belief, it was the
epoch of incredulity, it was the season
of light, it was the season of darkness,
it was the spring of hope, it was the
winter of despair, we had everything
before us, we had nothing before us,
we were all going direct to heaven, we
were all going direct the other way—in
short, the period was so far like the
present period, that some of its
noisiest authorities insisted on its
being received, for good or for evil, in
the superlative degree of comparison
only.
#CRC2014 @LoveStats
Based on any advertising, such as TV
commercials, roadside billboards, banners
on internet websites, or other similar forms
of advertising, please list in the textbox
below the products or types of products that
have you yourself have seen publicly
advertised between January 1, 2013 and
December 31, 2013. (Please type your
answer in the box below.)
#CRC2014 @LoveStats
Based on any advertising, such as TV
commercials, roadside billboards,
banners on internet websites, or other
similar forms of advertising, please list
in the textbox below the products or
types of products that have you
yourself have seen publicly advertised
between January 1, 2013 and
December 31, 2013. (Please type your
answer in the box below.)
What products did you
see advertised last
year?
What products did you
see advertised last
week?
#CRC2014 @LoveStats
Survey method
• Test n=530
• Control n=530
• US Census Rep
• Non-branded computer and
video gaming survey
• 20 minute survey
#CRC2014 @LoveStats
Traditional Humanized
What is your favorite video/computer
game?
If you could only play one video game
for the rest of your life, what game
would you choose?
Have you ever lined up at a store so
you could be among the first to
purchase a new video/computer game?
Are you one of those people who lines
up at stores to be the first to buy a new
game?
How much do you agree or disagree
with these statements about gaming?
Love 'em or hate 'em, what do you think
about video games?
What is your opinion about each of the
following television shows?
Awesome or terrible, how do you rate
these TV shows?
Is there any other information you
would like to share with us regarding
your opinions and use of
video/computer games?
Sharing is caring! Is there anything else
you'd like to share about gaming?
#CRC2014 @LoveStats
Data Quality
#CRC2014 @LoveStats
Speeding
• Failed if the survey was completed in the slowest
5% of completion times
0
1
2
3
4
5
Traditional Survey (%) Humanized Survey (%)
Speeding
#CRC2014 @LoveStats
Straightlining
• Failed if data demonstrated straightlining to a half
positive, half negative 8 question grid
0
5
10
15
20
Traditional Survey (%) Humanized Survey (%)
Straightlining
#CRC2014 @LoveStats
Open Ends
• Failed if less than 10 characters were provided in
response to a request for “3 reasons why…”
0
5
10
Traditional Survey (%) Humanized Survey (%)
Open Ends
#CRC2014 @LoveStats
Red Herring
• Failed if two fake computer games were selected
0
1
2
3
4
5
Traditional Survey (%) Humanized Survey (%)
Red Herring
#CRC2014 @LoveStats
Survey Evaluation
1.0
2.0
3.0
4.0
5.0
Too long Too complex Fun to answer Wouldrecommend to
my friendsTraditional Wording Fun Wording
Agree
Disagree
#CRC2014 @LoveStats
Responder Opinions Traditional Survey Humanized Survey
It was short and nice:) nice short survey not enough points though
I found this to be an interesting
survey. Thank you.
I love surveys and quizzes, and this one was very
interesting, because i like sharing things about myself.
the survey should be worth more
points considering how long the
survey was
very original method of conducting a survey!
NA This was one of the Best surveys I have done!!!
No loved it.. funny and quick
Good "May the survey FORCE be with you." Was a nice
hilariously awesome thing to see. I laughed a lot. o3o
This survey was enjoyable. this is an awesome survey. I thoroughly enjoyed myself
while taking it.
i dont know. good job, sir! If i had a fedora, i'd tip it in your direction.
It was good survey I laughed out loud the way some questions were phrased, I wanted to say this is the most refreshing survey ive taken I had a lot of fun
#CRC2014 @LoveStats
Survey Results
#CRC2014 @LoveStats
Purchase Intent
0%
10%
20%
30%
40%
Not at all Somewhat Very Extremely
Raw - Traditional Raw - Humanized
#CRC2014 @LoveStats
What is a fair price?
0%
10%
20%
30%
$0 $1 to $14 $15 to $19 $20 to $29 $30 to $39 $40 to $49 $50 or more
Raw - Traditional Raw - Humanized
#CRC2014 @LoveStats
Rating Ads
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
VUnlikely
SUnlikely
Neutral S Likely V Likely
Traditional Humanized
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
VUnlikely
SUnlikely
Neutral S Likely V Likely
Traditional Humanized
0.0%
10.0%
20.0%
30.0%
40.0%
VUnlikely
SUnlikely
Neutral S Likely V Likely
Traditional Humanized
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
VUnlikely
SUnlikely
Neutral S Likely V Likely
Traditional Humanized
On the package On the products
Before the game On the clothes
#CRC2014 @LoveStats
Implementing
the results
#CRC2014 @LoveStats
There is no perfect question
Are you the primary
grocery shopper in
your household?
Which of the
following statements
best describes your
role in
buying groceries for
your household?
Are you responsible
for most of
the grocery
shopping for your
household?
Do you normally do
most of the grocery
shopping for your
household?
#CRC2014 @LoveStats
• Agree Strongly
• Agree Somewhat
• Neutral
• Disagree Somewhat
• Disagree Strongly
• Love
• Like
• Neutral
• Dislike
• Hate
Ask for simpler wording
#CRC2014 @LoveStats
Which of the following do you enjoy?
• Biking
• Cooking
• Hiking
• Reading
Which of the following
do you enjoy?
• Bike
• Cook
• Hiking
• Read
Don’t compromise on grammar
Let’s eat Grandma.
Let’s eat, Grandma.
Punctuation saves
lives.
#CRC2014 @LoveStats
For each of the following descriptions of shopping behaviors, please indicate whether the description is
highly characteristic, somewhat characteristic, slightly characteristic, or not characteristic at all of
you when you visit a membership-only
warehouse club store.
How descriptive are these
characteristics when you visit a
membership-only warehouse club
store.
Ask for shorter questions
#CRC2014 @LoveStats
Ask for a bit of humour
Religion
Politics Sex
Violence Race
Pets Movies
Neutral jokes
Common slang
#CRC2014 @LoveStats
Answer options can be fun too
What we usually say: What you can say:
Every day Every day. I think I'm
addicted!
More than 10 hours More than 10 hours. You
can't make me stop!
No one, you normally
play by yourself
Me, myself, and I
Never Never. I have better
things to do!
Zero Zero, zilch, nada!
#CRC2014 @LoveStats
Thank you
for your
participation
You’ve been a great help. Thanks so
much!
Say please and thank you
#CRC2014 @LoveStats
Let the people speak What we usually say: What you can say:
Please describe 3 reasons why that
is your favorite video/computer
game.
Give me 3 good reasons why!
Is there any other information you
would like to share with us regarding
your opinions and use of
video/computer games? (Optional)
Sharing is caring! Is there anything
else you'd like to share about
gaming? (Answer if you wish)
Compared to other surveys you have
taken, what is your opinion about
this survey? (Please select one in
each row)
What did you think of this survey?
(Please select one in each row)
This is the last page of questions. Is
there any other information you
would like to share about this
survey? (Optional)
This is the end, my friend. Is there
anything about this survey you'd care
to share? (Answer if you wish)
#CRC2014 @LoveStats
9/23/2014
27
Thank you!
Annie Pettit
Chief Research Officer
Peanut Labs
@LoveStats on twitter
facebook.com/anniepettit
ca.linkedin.com/in/anniepettit/
: )
180 Montgomery Street, Suite 1700 - San Francisco, CA 94104