the roots of total survey design lars lyberg stockholm university qmms seminar leinsweiler, nov 7-9,...
TRANSCRIPT
![Page 1: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/1.jpg)
The Roots of Total Survey Design
Lars LybergStockholm University
QMMS Seminar Leinsweiler, Nov 7-9, 2010
![Page 2: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/2.jpg)
Early thinkers Hansen and colleagues, U.S.
Bureau of the Census Deming, U.S. Bureau of the Census
and consultant Kish, University of Michigan Dalenius, Statistics Sweden and
Stockholm University
![Page 3: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/3.jpg)
What were they thinking about? Nonsampling errors Balancing errors and costs Design criteria The limitations of sampling theory Standards Similarities between survey
implementation and the assembly line
![Page 4: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/4.jpg)
4
Deming (1944) “On Errors in Surveys”
American Sociological Review! First listing of sources of problems,
beyond sampling, facing surveys The 13 factors
![Page 5: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/5.jpg)
Deming’s 13 factors
The 13 factors that affect the usefulness of a survey
-To point out the need for directing effort toward all of them in the planning process with a view to usefulness and funds available
-To point out the futility of concentrating on only one or two of them
-To point out the need for theories of bias and variability that correlate accumulated experience
![Page 6: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/6.jpg)
6
![Page 7: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/7.jpg)
Their difficult position They had to promote Neyman’s theory But his theory basically assumes very
small nonsampling errors They were in a first-things-first situation They promoted vigorous controls
hopefully leading to small biases They discussed what a Bayesian
approach might offer
![Page 8: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/8.jpg)
Lines of thought I “There is as yet no universally
accepted ‘survey design formula’ that provides a solution to the design problem (Dalenius 1967)
That’s why textbooks devote little space to design
Important to control specific error sources
![Page 9: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/9.jpg)
Lines of thought II The U.S. Bureau of the Census is a
statistical factory. The main product is statistical tables (Deming and Geoffrey 1941)
Concentration on QC of error sources, evaluation, and survey models
Disentangling the design process
![Page 10: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/10.jpg)
Lines of thought III Hansen-Hurwitz-Pritzker 1967
Take all error sources into account Minimize all biases and select a minimum-
variance scheme so that Var becomes an approximation of (a decent) MSE
The zero defects movement that later became Six Sigma
Dalenius 1969 Total survey design
![Page 11: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/11.jpg)
The design process Criterion of effectiveness: Minimum MSE
per unit of cost while meeting other requirements such as timeliness of results (not just minimum variance)
Good survey design calls for reasonably effective control of the accuracy through appropriate specifications for survey procedures and adequate control of the operations, i.e. proper design of the total system
![Page 12: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/12.jpg)
Mean squared error (MSE) MSE=Var+B2+(Relevance
error)2+Interaction MSEZ(y)=E(y-Y)2+(Y-X)2+(X-Z)2
+2(Y-X)(X-Z)Z is the ideal goal, X is the defined goal,
and y is the actual result Hansen-Hurwitz-Pritzker call them
requirements (Z), specifications (X) and operations (y)
![Page 13: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/13.jpg)
Design issues X-Z is crucial in the design
situation Do we want an approximate
solution to the right problem or an exact solution to the wrong problem?
![Page 14: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/14.jpg)
The design approach (Dalenius and Hansen et al) Specify the ideal goal Z Analyze the survey situation
(financial, methodological and information resources)
Construct a small number of alternative designs
Evaluate the alternatives by reference to associated MSE and costs
![Page 15: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/15.jpg)
The design approach (cont’d) Make a decision
Use one of the alternatives Use a modification of one of them Do not conduct the survey
Develop the administrative design Feasibility The signal system A self-contained design document (tree) Plan B
![Page 16: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/16.jpg)
What does this tell us?
All error sources should be taken into account
There is very little process talk such as the need for CQI
However, the common situation was: no process view, no controls
Concern about costs and effectiveness of all these controls
The user is a somewhat distant player
![Page 17: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/17.jpg)
The user The user was hiding under terms such
as subject matter problem, study purpose or “the four key functions of a statistical system” (reporting, analytic, consulting, research) Tukey 1949
But there were federal statistics users conferences in the U.S. from 1957. Dalenius provides more than 200 references on users in a 1967 ISI paper
![Page 18: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/18.jpg)
Who identifies the requirements? Usually seen as one fictive person
An official An administrator A statistician acting as a subject-
matter specialist Requirements define the population,
types of measurement, time dimensions and statistics needed
![Page 19: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/19.jpg)
The designer’s role vis-à-vis the requirements To critique the suggested
requirements To suggest QC procedures, construct
dummy tables to check the decision-making and perform sensitivity analysis
To act as the devil’s advocate and discuss specific result interpretations with the user
![Page 20: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/20.jpg)
Kish’s contributions The neo-Bayesian view
Appreciates the literature by Schlaifer, Ericson, Edwards, Lindman and Savage on Bayesian methods in survey sampling and psychometrics
For instance, judgment estimates of measurement biases may be combined with sampling variances to construct more realistic estimates of the total survey error
![Page 21: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/21.jpg)
More from Kish Experiments and sample surveys
might not be sufficient. Other investigations “collecting data with considerable care and control” but without randomization and probability sampling might be necessary.
![Page 22: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/22.jpg)
Kish’s view on design Multipurpose is great from an economical
point of view. If one principal statistic can be identified
that alone can decide the design If a small number of principal statistics
can be identified a reasonable design compromise is possible
If statistics are too disparate a joint design might not be possible
![Page 23: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/23.jpg)
Kish on economic design Requires joint consideration of sampling
and nonsampling errors Sometimes demands prior or pilot
studies of sufficient size Requires information about unit
variance Emphasizes a small total error Appreciates the fact that a reduction of
one source might increase total error
![Page 24: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/24.jpg)
Examples of decisions Frame needs updating? Reference period? Acceptable respondent rules? Number of callbacks? Allocation of callbacks? How much and what kind of editing? Mix of modes?
![Page 25: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/25.jpg)
Kish summed up Get a good balance between different
error sources We need to know how error structures
behave under different design alternatives Relevant information should be recorded
during implementation (paradata) Many practical constraints The multipurpose nature calls for a
compromise
![Page 26: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/26.jpg)
Hansen, Dalenius and colleagues on standards General standards
Measurable survey plan, self-contained plan, replications should generate similar results, cost-efficient, sufficiently simple plan
Standards for error control Relevance control, control of accuracy (should
be dominated by variance terms) Minimum performance standards Check that standards yield the results expected
![Page 27: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/27.jpg)
Hansen, Dalenius and colleagues summed up One should be guided by common sense,
experience and theory Design and execution is a management and
systems analysis problem A survey is an economic production process Survey goals must be identified Standards must be dynamic End the practice that sampling error is viewed
as the total error They predicted the CASM movement
![Page 28: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/28.jpg)
More from Hansen, Dalenius and colleagues The examination of design alternatives is costly
and time-consuming There is a risk of overcontrol and inadequate
control. Consequences of large errors must guide any relaxation but they don’t talk about CQI
One might have to compromise relevance to get controllable measurements or abstain from the survey
Keep bias near zero and allow variance at expected levels
![Page 29: The Roots of Total Survey Design Lars Lyberg Stockholm University QMMS Seminar Leinsweiler, Nov 7-9, 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062404/5515f5dc550346cf6f8b562d/html5/thumbnails/29.jpg)
What happened? Still no “design formula” General design principles exist for some areas Still a concentration on some error sources
more than others CASM happened We got standards The TSE paradigm accepted but has some
promotional problems Many of the early thoughts were just that, very
little practice, but still useful