New Dimensions of Quality in Online Panels

Jacqueline LorchVice President, Global Knowledge Management

Survey Sampling International

The World Is Moving On…

Sampling

Questionnaire Design

Interpretation

Fieldwork

Why Does It Matter?

What Are The Key Quality Elements?

A broadly-recruited, representative, well-managed pool of respondents

Respondents who respond honestly and conscientiously:

What guarantees are there to guard against bad data, i.e. respondent cheating or not concentrating/caring in their responses… ?

A well-designed survey instrument

How Do Respondents Fit In?

Google

“Market Research” 464 million entries

“Market Research Respondents” 2 million entries

What Is A ‘Professional Respondent’?

Some assumptions:

Quality of responses will be lower

Motivated to maximize incentives, not by intrinsic interest

Do too many surveys – become conditioned

But what definition?

Someone who gets paid?

“I am an IT Director with 7 wives…”

Honesty Is The Issue

Smith and Hofma Brown, Harris Interactive

New Typology Of “Bad” Respondents

Hyperactive Respondents

Too many surveys, too many panels

Fraudulent Respondents

Misrepresent themselves

Inattentive Respondents

Don’t put thought into answers

Conditioned Respondents

Have learned from past surveys

Hyperactive Respondents

Do busy panelists provide bad data?

Do the most responsive panelists take the surveys the fastest?

Are the fastest surveys the ‘worst’?

Hyperactive Respondents

US SurveySpot panel, 3 month internal tracker

Two groups:

– SurveySpot only

– One or more other panels

No difference in average time taken by each group

Similar answers on motivation for joining:

Motivated by chance to influence decisions

– SurveySpot-only: 50%

– More than one panel membership: 47%

Fraudulent Respondents

How can you tell if someone is who they say they are?

How can you stop multiple panel memberships using different identities?

How can you tell if someone is just making up the answers?

Who Are You Today?

Ask me something only I would/should know…..

Catch The Cheat…

Respondents…

Who have 23 different ailments

Who report using non-existent brands

Whose education doesn’t match their profession

Inattentive Respondents

Fatigue leads them to skip questions

Don’t pay attention to instructions

May be inevitable after a certain length of survey

Related to interview length

5% baseline of inattention?

Length of interview, minutes 14 19 21

Items 15 5 15

Skipped all 5% 8% 11%

Inattentive Respondents

Respondents Speak Out

Repetitive questions “sorry this survey was just too long.”

“Sometimes it becomes so repetitive you say, ‘to hell with it, I don’t need this.’”

“You think you are about done and the same questions start all over again.”

Time use > 7300 hours = 4 hours of sleep a night

Time use > 8760 hours = no sleep, ever.. more hours than in a year

5% baseline of inattention?

A really tough question set?

Length of interview, minutes 14 19 21

>7300 hours 5% 6% 6%

>8760 hours 3% 4% 4%

Inattentive Respondents

Respondents Speak Out

“…absolutely ridiculous… [questions like] ‘if this

pizza was a person.’ ”

“Why is this bottled water like your favorite pet?”

“In filling out this survey it asked when I would buy a new house. I said “never”. The next several questions were regarding my new house and it required an answer…So I quit the survey and didn’t finish it.”

Conditioned Respondents

Only give the answers they do because of what they have learned from previous surveys

Or change behavior as a result of information from surveys

Have no redeeming features

But do such people exist…..?

TNS Experiment

3 groups of respondents

High frequency – interviewed 5 times (1.2.3.4.5)

Medium frequency – interviewed 3 times (1.3.5)

Low frequency – interviewed 2 times (1.5)

+ Control group at wave 5

Same questionnaire

Survey frequency not yet released by TNS

UK, France, Germany

n = 1202 (control = 1470)

Product Usage

High Low

Product Purchase

High Low

Brand Awareness

High Low

Conditioned Respondents

Evidence is hard to find

Maybe surveys aren’t quite so important to respondents as they are to researchers!

Could we be over-reacting and losing good respondents?

Respondents Speak Out

“I would like to know why it is that practically every time I give my age I am refused the access to the survey? Is there something wrong with being 75 and in good health, mentally and physically?”

Why do I spend 10 minutes answering questions on one of your surveys before I am given message ‘Sorry, you didn't qualify for this survey’.

“I have not been able to take several [most] surveys because I work in the grocery industry. I don’t think it is fair. I AM A CONSUMER TOO … It is not like I work for a company that makes or sells one brand.”

Respondents Speak Out

“…ability of the human eyes to see fonts of microscopic size. What’s with this survey? The font size was like .002.”

“I said yes before; why are you going back and asking me the same question.”

“Right to the point”…“if they say it’s 5 minutes, it’s 5 minutes.”

“By and large ask sensible questions in a straightforward way.”

What Needs To Happen?

Partnership…common terms and definitions

Avoid red herrings

We are not in the business of supplying bad panelists

Survey design is critical

We can’t do it alone