Apples to Oranges or

Gala vs. Golden Delicious?

Comparing Data Quality of

Nonprobability Internet Samples to

Low Response Rate Probability

Samples

David Dutwin, Ph.D., SSRS

Trent D. Buskirk, Ph.D., MSG AAPOR, 2016

1.

Background

What’s the Problem, Exactly?

Non-probabilistic data sources can have self-selection bias

over and above what probability panels might have:

As such, considerable variance in the universe of web

panels:

* Regardless of Hispanicity

8%

32%

23% 22%

10%

0%

5%

10%

15%

20%

25%

30%

35%

Panel 1 Panel 2 Panel 3 Panel 4 Panel 5

Percent of Web Panelists 18-24 from 5 Leading Convenience Panels

64%69%

80%69%

62%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Panel 1 Panel 2 Panel 3 Panel 4 Panel 5

Percent of Web Panelists White* from 5 Leading Convenience Panels

Who are the Web Panelists Anyway?

67% 64%

0%

20%

40%

60%

80%

Panelist USA

White Non-Hispanic

13% 13%

0%

10%

20%

30%

Panelist USA

Age 18-29

60%52%

0%

20%

40%

60%

80%

Panelist USA

Female

33%30%

0%

10%

20%

30%

40%

Panelist USA

Democrat

33% 34%

0%

10%

20%

30%

40%

Panelist USA

2 Person HH

50% 53%

0%

20%

40%

60%

80%

Panelist USA

Married

Well, they look a lot like Americans…

Who are the Web Panelists Anyway?

Except that they don’t…

10%

19%

0%

10%

20%

30%

Panelist USA

Retired

60%50%

0%

20%

40%

60%

80%

Panelist USA

H.C Thru a Health Plan

78%

40%

0%

20%

40%

60%

80%

Panelist USA

On Internet Multiple Times/Day

87%

64%

0%

20%

40%

60%

80%

100%

Panelist USA

Own a Smartphone

24%

36%

0%

20%

40%

Panelist USA

Age 65+

37%29%

0%

10%

20%

30%

40%

50%

Panelist USA

College Degree

Non-Probability seductively

cheaper

Non-probability vary in

execution, recruitment and

quality Pew Research Report, 2016

Methods based on modeling,

weighting and matching

continue to emerge to improve

quality of estimates from non-

prob samples. Terhanian et al., 2016

Dever et al., 2015

DiSogra et al., 2015

Rivers and Bailey, 2009

Dutwin and Buskirk, 2016

Probability Samples cost more

When compared head to head,

estimates from probability

samples tend to be more

accurate than those from non-

prob samples Callegaro et al., 2014;

Yeager et al., 2011

Krosnick and Chang, 2009

Walker et al., 2009

Response rates continue to

decline

Cost vs. Quality?

Our main Research Questions

How does the quality of non-probability samples

compare to that of low-response rate probability

samples?

Can we improve the quality of estimates from non-

probability samples using alternative adjustment

methods like calibration, propensity weighting or sample

matching?

2.

Data and Methods

Data Sources

Dual-Frame RDD Telephone Sample 1 n = 107,347; ~9% Response Rate.

Non-probability Web Panel 1 n = 82,478; Response Rate unknown.

Dual-Frame RDD Telephone Sample 2 n = 29,153; ~12% Response Rate.

Non-probability Web Panel 2 n = 61,782; Response Rate unknown.

All samples selected and fielded between

October 2012 thru 2014

1st

Sample

Set

2nd

Sample

Set

Methods for Adjusting Non-Probability samples

Base Weight = 1

other than Dual-

Frame which gets

single frame

estimator (Best

and Buskirk,

2012).

Raking to

Education, Region,

Gender, Age,

Race/Ethnicity

No Trimming

Raking/

Calibration

Conducted only in NP-

Panel #2 which

included webographics

Logistic Regression

Model: Education,

Region, Gender, Age,

Race/Ethnicity, Metro

Status, # of Adults,

and Webographics

Tested both raw

scores and a weighting

class (5) variant

Propensity

Adjustments

Sample

MatchingBased on pairing

non-probability

cases with

members of a

probability sample

Sample matches

based on similarity

across core set of

common variables

(Rivers and Bailey,

2009)

Applied to both NP

panel samples

The Matching Process…

Isn’t Fool Proof…

Creating Matched Samples

The matching algorithm uses a

collection of categorical variables from

each case in the Prob. sample and

computes a similarity index defined as

the simple matching coefficient (SMC)

to each case in the NP sample.

The matched case is randomly

selected from among those identified

as the most similar.

See: http://bit.ly/1Fb2Jhs for specific

details of computing the SMC for

categorical variables.

212

Generating the Matched Sample

An 3.5% SRS of the Adults contained in the 2013/2014 CPS Public

Release Data file were matched to sampled units in the combined

NP Panel 1/Panel 2, respectively sample based on 8 common

demographic variables including:

Region (North, South, East, West)

Male

Age Group (18-29, 30-49, 50-64 and 65+)

Race Group (White (NH), Black (NH), Hispanic, Other (NH))

Education Level (<HS, HS, Some College, BS, BS+)

Own/Rent (Own, Rent/Board)

Marital Status (Married, Single, Partnered, Divorced/ Widow/ Separated)

Employment Status (Currently Employed or not)

3.

Outcomes and

Metrics

Primary Metrics

The mean absolute

bias (MAB) –

computed as the

arithmetic mean of

absolute value of the

difference between

the table estimate and

the corresponding

benchmark estimate the mean is taken

over the total number

of estimates within

the variable set

Absolute Bias

The standard

deviation of the

absolute biases

computed from

each variable set

was also

computed. Provides a

sense of the

variability in the

level of biases.

The overall

average MAB –

computed as

the mean of the

12 MAB

statistics

computed

across the 12

variable sets for

each sample

St. Deviation

of Biases

Overall

Average MAB

Key Outcomes We Consider

Common set of survey variables across the sources of data

include household and person-level demographics

External benchmarks for media related information contained

in the main survey source are not commonly available

Given this scenario, we will focus our evaluation and

computation of bias metrics on distributions of one

demographic variable within levels of a second

What is the distribution of Education within each level of Race

What is the distribution of Race within each level of Education

The reference/benchmark values are computed using the 1

Year PUMS Data from the 2012 American Community Survey.

Evaluated Outcome Variable SetsSpecific Demographic Variable Sets of interest include:

Education (5 levels) within Race (4 levels) and and Race within Education

Education (5 levels) within Age-group (4 levels)

and Age-group within Education

Education (5 levels) within Region (4 levels) and Region within Education

Age-group (4 levels) within Race (4 levels) and Race within Age-group

Age-group (4 levels) within Region (4 levels) and Region within Age-group

Race (4 levels) within Region (4 levels) and Region within Race

Computing the Primary Metrics

4.

Results

MAB Statistics for Demographic Cross-Tabulations

1st

Sample

Set

Unweighted Overall Average Biases

1st

Sample

Set

MAB Statistics for Demographic Cross-Tabulations

2nd

Sample

Set

Unweighted Overall Average Biases

2nd

Sample

Set

Unequal Weighting Effects

Unequal Weighting Effects

Telephone 2: 1.21

NP Panel 2 Rake: 2.83

NP Panel 2 Propensity: 6.35

NP Panel 2 Propensity and Rake:

5.43

Unequal Weighting Effects

Telephone 1: 1.36

ABS: 1.74

NP Panel 1: 2.71

NP Panel Matched and Raked:

1.18

1st

Sample

Set

2nd

Sample

Set

Variability in Absolute Bias Measures

1st

Sample

Set

Variability in Absolute Bias Measures

2nd

Sample

Set

5.

Conclusions and

Discussion

Discussion

Methods for improving the quality of nonprobability panels continue to

be made including extensive use of modelling/optimization methods for

selecting candidate variables for weighting (Terhanian et al., 2016)

While we saw that matched samples (based on a simple matching

coefficient) tended to move the absolute bias measures downward,

compared to unweighted and unmatched nonprobability samples, they

still produced estimates with inherently more bias with more variability

than probability samples.

More work is needed to better understand how to optimize the matching

process including: How to incorporate a mixture of categorical and continuous variables

Optimal combination of matching and raking variables

Incorporation of sampling weights into the matched process

Optimal relative size of non-probability and probability samples used for

matching.

What does a 9% response rate get you that a

web panel cannot?

Unweighted, substantially less bias

Weighted, significantly, but not substantially, less bias

A much lower design effect from weighting

Generally less variability in the absolute biases

That said, matched samples attain the lowest design

effect of any weighting

And that said, matched samples “close” to weighted

telephone in terms of lower bias and lower variability in

the absolute biases….but still substantially inferior

And…there is still the issue of cost.

Thank you!

References Available Upon Request

Please contact us with questions

TBuskirk@m-s-g.com | 314-695-1378 |

DDutwin@ssrs.com | 484-840-4406 |

Generating the Matched Sample

SlidesCarnival icons are editable shapes.

This means that you can:

● Resize them without losing quality.

● Change fill color and opacity.

Isn’t that nice? :)

Examples:

Extra graphics