do we still need probability sampling in surveys? robert m. groves university of michigan and joint...
TRANSCRIPT
![Page 1: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/1.jpg)
Do We Still Need Probability Sampling in Surveys?
Robert M. Groves
University of Michigan and
Joint Program in Survey Methodology, USA
![Page 2: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/2.jpg)
Outline
• The total survey error paradigm in scientific surveys
• The decline in survey participation
• The rise of internet panels
• The “second era” of internet panels
• So... do we need probability sampling?
![Page 3: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/3.jpg)
Outline
• The total survey error paradigm in scientific surveys
• The decline in survey participation
• The rise of internet panels
• The “second era” of internet panels
• So... do we need probability sampling?
![Page 4: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/4.jpg)
The Ingredients of Scientific Surveys
• A target population
• A sampling frame
• A sample design and selection
• A set of target constructs
• A measurement process
• Statistical estimation
![Page 5: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/5.jpg)
Deming (1944) “On Errors in Surveys”
• American Sociological Review!
• First listing of sources of problems, beyond sampling, facing surveys
![Page 6: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/6.jpg)
![Page 7: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/7.jpg)
Comments on Deming (1944)
• Includes nonresponse, sampling, interviewer effects, mode effects, various other measurement errors, and processing errors
• Includes nonstatistical notions (auspices)• Includes estimation step errors (wrong
weighting) • Omits coverage errors• “total survey error” not used as a term
![Page 8: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/8.jpg)
Sampling Text Treatment of Total Survey Error
• Kish, Survey Sampling, 1965– 65 of 643 pages on various errors, with
specified relationship among errors– Graphic on biases
![Page 9: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/9.jpg)
Sampling Biases
Frame biases
“Consistent” Sampling Bias
Constant Statistical Bias
Nonsampling
Biases
Noncoverage
NonresponseNonobservation
Field: data collection
Office: processingObservation
![Page 10: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/10.jpg)
Total Survey Error (1979)Anderson, Kasper, Frankel, and Associates
• Empirical studies on nonresponse, measurement, and processing errors for health survey data
• Initial total survey error framework in more elaborated nested structure
![Page 11: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/11.jpg)
Total Error
VariableError
Sampling
Nonsampling
Field
Processing
Bias
Nonsampling
Observation
Field
Processing
Sampling
Frame
Consistent
Nonobservation
Noncoverage
Nonresponse
![Page 12: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/12.jpg)
Survey Errors and Survey Costs (1989), Groves
• Attempts conceptual linkages between total survey error framework and– psychometric true score theories– econometric measurement error and selection bias
notions
• Ignores processing error• Highest conceptual break on variance vs. bias• Second conceptual break on errors of
nonobservation vs. errors of observation
![Page 13: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/13.jpg)
Coverage Nonresponse Sampling Interviewer Respondent Instrument Mode
Coverage Nonresponse Sampling Interviewer Respondent Instrument Mode
Errors ofNonobservation
ObservationalErrors
Bias
Errors ofNonobservation
ObservationalErrors
Variance
Mean Square Error
construct validitytheoretical validityempirical validityreliability
criterion validity - predictive validity - concurrent validity
![Page 14: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/14.jpg)
Nonsampling Error in Surveys (1992), Lessler and Kalsbeek
• Evokes “total survey design” more than total survey error
• Omits processing error
![Page 15: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/15.jpg)
Components of Error Topics
Frame errors Missing elements
Nonpopulation elements
Unrecognized multiplicities
Improper use of clustered frames
Sampling errors
Nonresponse errors Deterministic vs. stochastic view of nonresponse
Unit nonresponse
Item nonresponse
Measurement errors Error models of numeric and categorical data
Studies with and without special data collections
![Page 16: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/16.jpg)
Introduction to Survey Quality, (2003), Biemer and Lyberg
• Major division of sampling and nonsampling error
• Adds “specification error” (a la “construct validity”)
• Formally discusses process quality
• Discusses “fitness for use” as quality definition
![Page 17: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/17.jpg)
Sources of Error Types of Error
Specification error Concepts
Objectives
Data element
Frame error Omissions
Erroneous inclusions
Duplications
Nonresponse error Whole unit
Within unit
Item
Incomplete Information
Measurement error Information system
Setting
Mode of data collection
Respondent
Interview
Instrument
Processing error Editing
Data entry
Coding
Weighting
Tabulation
![Page 18: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/18.jpg)
Survey Methodology, (2004) Groves, Fowler, Couper, Lepkowski, Singer,
Tourangeau
• Notes twin inferential processes in surveys– from a datum reported to the given construct
of a sampled unit– from estimate based on respondents to the
target population parameter
• Links inferential steps to error sources
![Page 19: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/19.jpg)
Construct Inferential Population
Measurement
Response
Target Population
Sampling Frame
Sample
Validity
Measurement Error
Coverage
Error
Sampling
Error
Measurement Representation
Respondents
Nonresponse
ErrorEdited Data
ProcessingError
Survey Statistic
The Total Survey Error Paradigm
![Page 20: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/20.jpg)
Summary of the Evolution of “Total Survey Error”
• Roots in cautioning against sole attention to sampling error
• Framework contains statistical and nonstatistical notions
• Most statistical attention on variance components, most on measurement error variance
• Late 1970’s attention to “total survey design”• 1980’s-1990’s attempt to import psychometric
notions• Key omissions in research
![Page 21: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/21.jpg)
5 Myths of Survey Practice that TSE Debunks
1. “Nonresponse rates are everything”2. “Nonresponse rates don’t matter”3. Give as many cases to the good
interviewers as they can work4. Postsurvey adjustments eliminate
nonresponse error5. Usual standard errors reflect all sources
of instability in estimates (measurement error variance, interviewer variance, etc.)
![Page 22: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/22.jpg)
Outline
• The total survey error paradigm in scientific surveys
• The decline in survey participation
• The rise of internet panels
• The “second era” of internet panels
• So... do we need probability sampling?
![Page 23: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/23.jpg)
Response Rates
• In most rich countries response rates on household and organizational surveys are declining
• deLeeuw and deHeer (2002) model a 2 percentage point decline per year
• Probability sampling inference is unbiased from nonresponse with 100% response rate
![Page 24: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/24.jpg)
• Recent studies challenge a simple link between response rates and nonresponse error
• Reading Keeter et al. (2000), Curtin et al. (2000), Merkle and Edelman (2002) suggests response rates don’t matter
• Standard practice urges maximizing response rates
What’s a practitioner to do?
![Page 25: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/25.jpg)
Mismatches between Statistical Expressions for Nonresponse Error
and Practice
pyyy
n
myy yp
nmrnr
)(
p ,propensity
response the and y,variable, survey the between covariance
where
yp
![Page 26: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/26.jpg)
What does the Stochastic View of Response Propensity Imply?
• Key issue is whether the influences on survey participation are shared with the influences on the survey variables
• Increased nonresponse rates do not necessarily imply increased nonresponse error
• Hence, investigations are necessary to discover whether the estimates of interest might be subject to nonresponse errors
![Page 27: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/27.jpg)
Assembly of Prior Studies of Nonresponse Bias
• Search of peer-reviewed and other publications• 47 articles reporting 59 studies • About 959 separate estimates (566
percentages)– mean nonresponse rate is 36%– mean bias is 8% of the full sample estimate
• We treat this as 959 observations, weighted by sample sizes, multiply-imputed for item missing data, standard errors reflecting clustering into 59 studies and imputation variance
![Page 28: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/28.jpg)
Percentage Absolute Relative Bias
mean sample full unadjusted the is y
mean respondent unadjusted the is y where
n
r
n
nr
y
yy )(*100
![Page 29: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/29.jpg)
Percentage Absolute Relative Nonresponse Bias by Nonresponse Rate for 959
Estimates from 59 Studies
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80Nonresponse Rate
Per
cen
tag
e A
bso
lute
Rel
ativ
e B
ias
![Page 30: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/30.jpg)
30
1. Nonresponse Bias Happens
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80Nonresponse Rate
Per
cen
tag
e A
bso
lute
Rel
ativ
e B
ias
![Page 31: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/31.jpg)
31
2. Large Variation in Nonresponse Bias Across Estimates Within the Same
Survey, or
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80Nonresponse Rate
Per
cen
tag
e A
bso
lute
Rel
ativ
e B
ias
![Page 32: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/32.jpg)
32
3. The Nonresponse Rate of a Survey is a Poor Predictor of the Bias of its Various
Estimates (Naïve OLS, R2=.04)
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80
Nonresponse Rate
Pe
rce
nta
ge
Ab
so
lute
Re
lati
ve
Bia
s o
f R
es
po
nd
en
t M
ea
n
![Page 33: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/33.jpg)
Conclusions
• It’s not that nonresponse error doesn’t exist
• It’s that nonresponse rates aren’t good predictors of nonresponse error
• We need auxiliary variables to help us gauge nonresponse error
![Page 34: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/34.jpg)
A Practical Question
“What attraction does a probability sample have for representing a target population if its nonresponse rate is very high and its respondent count is lower than equally-costly nonprobability surveys?”
![Page 35: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/35.jpg)
Outline
• The total survey error paradigm in scientific surveys
• The decline in survey participation
• The rise of internet panels
• The “second era” of internet panels
• So... do we need probability sampling?
![Page 36: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/36.jpg)
A “Solution” to Response Rate Woes
• Web surveys offer a very different cost structure than telephone and face-to-face surveys– Almost all fixed costs– Very fast data collection
• But there is no sampling frame– Often probability sampling from large volunteer
groups
• Internet access varies across and within countries
![Page 37: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/37.jpg)
Access/Volunteer Internet Panels
• Massive change in US commercial survey practice, moving from telephone and mail paper questionnaires to web surveys
• Survey Sampling, a major supplier of telephone samples over the past two decades now reports that 80% of their business is web panel samples
• Some businesses do only web survey measurement
![Page 38: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/38.jpg)
The Method
• Recruitment of email ID’s from internet users– At survey organization’s web site– Through pop-ups or banners on others’ sites– Through third party vendors
• A June 15, 2008, Google search of “make money doing surveys” yields 19,300 hits– “make $10 in 5 minutes” www.SurveyMonster.com
![Page 39: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/39.jpg)
![Page 40: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/40.jpg)
40
There is a new industry– Greenfield Online
– Survey Sampling
– e-Rewards
– Lightspeed
– ePocrates
– Knowledge Networks
– Private company panels
– Proprietary panels
U.S. Online MR Spending
$0
$200
$400
$600
$800
$1,000
$1,200
$1,400
$1,600
$1,800
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
E
Mill
ion
s
Inside Research, 2007
Baker, 2008
![Page 41: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/41.jpg)
Reward Systems Vary
• Payment per survey
• Points per survey, yielding eligibility for rewards
• Points for sweepstakes
![Page 42: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/42.jpg)
Adjustment in Estimation
• Estimation usually involves adjustment to some population totals
• Some firms have propensity model-based adjustments– “proprietary estimation systems” abound
![Page 43: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/43.jpg)
Outline
• The total survey error paradigm in scientific surveys
• The decline in survey participation
• The rise of internet panels
• The “second era” of internet panels
• So... do we need probability sampling?
![Page 44: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/44.jpg)
September, 2007, Respondent Quality Summit
• Head of Proctor and Gamble market research1. Cites Comscore: 0.25% of internet users
responsible for 30% of responses to internet panels
2. Cites average number of panel memberships of respondents of 5-8
3. Presents examples of failure to predict behaviors
![Page 45: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/45.jpg)
45
The number of surveys taken matters.
78%
43%
60%
73%
38%
51%
66%
33%
46%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Like Product Intend to Buy Expected PurchaseFrequency
1-3 Surveys 4-19 Surveys 20+ Surveys
Coen et al., 2005 in Baker, 2008
![Page 46: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/46.jpg)
46
The Practical Indicators of “Quality”
• Cheating on qualifying questions• Internal inconsistencies• Overly fast completion• “Straightlining” in grids• Gibberish or duplicated open end responses• Failure of “verification” items in grids• Selection of bogus or low-probability answers• Non-comparability of results with non-panel
sample
Baker, 2008
![Page 47: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/47.jpg)
47
Panel response rates are in decline as panelists do more surveys.
54%59% 61%
69%
20% 18%
11%5%
0%
20%
40%
60%
80%
Web1 Web2 Web3 Web4
More than 15 Surveys Response Rate
MSI, 2005 in Baker, 2008
![Page 48: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/48.jpg)
Where are we now?
• An industry in turmoil
• Active study of correlates of low quality conducted by sophisticated clients
• Professional associations attempting to define quality indicators
![Page 49: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/49.jpg)
Outline
• The total survey error paradigm in scientific surveys
• The decline in survey participation
• The rise of internet panels
• The “second era” of internet panels
• So... do we need probability sampling?
![Page 50: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/50.jpg)
Access Panels and Inference
• Access panels have conjoined frame development and sample selection
• Without documentation of the frame development, assessment of coverage properties are not tractable
• Many use probability sampling from the volunteer set, but ignore this in estimation
![Page 51: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/51.jpg)
A Better Question
• Not “do we still need probability sampling?” but “can we develop good sampling frames with rich auxiliary variables?”
![Page 52: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/52.jpg)
Target Population
Sampling Frame
Sample
Respondents
Model-assisted
Randomizationtheory
Model-assisted
Target Population
Sampling Frame
Sample
Respondents
Model-assisted
?
![Page 53: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/53.jpg)
The Value of Probability Sampling From Well-defined Frames
• Randomization theory is the powerful linking tool between the sample and the frame
• Models of nonresponse adjustment are enhanced by auxiliary variables measured on respondents and nonrespondents
![Page 54: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/54.jpg)
The Role of Probability Sampling in this Context
• Probability sampling has low marginal costs within a defined sampling frame
• Probability sampling offers stratification benefits
• A sampling frame with rich auxiliary variables can improve stratification effects
Access panels should strive for well-defined frame development
![Page 55: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/55.jpg)
Speculation
• As adjustment for nonresponse becomes more important,– Richness of auxiliary variables is primary– Coverage of population becomes relatively
less important
• Hence, frame data and field observations on nonrespondents and respondents are valued
![Page 56: Do We Still Need Probability Sampling in Surveys? Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA](https://reader031.vdocuments.mx/reader031/viewer/2022032802/56649dea5503460f94ae5791/html5/thumbnails/56.jpg)
Outline
• The total survey error paradigm in scientific surveys
• The decline in survey participation
• The rise of internet panels
• The “second era” of internet panels
• So... do we need probability sampling?