basic question evaluation
DESCRIPTION
Kirsten Miller gave a presentation about survey question design based on her work at the Question Design Research Lab at the National Center for Health Statistics. Her talk was given at DePaul University in Chicago, Illinois on February 13, 2012. This event was sponsored by the DePaul College of Liberal Arts and Social Sciences, the Social Science Research Center, and the Department of Sociology. Audio from the presentation can be heard here: http://is.gd/ssrc_kmillerTRANSCRIPT
![Page 1: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/1.jpg)
Kristen Miller, Ph.D. Question Design Research
Office of Research and Methodology
National Center for Health Statistics
Question Evaluation at the National Center for Health Statistics
![Page 2: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/2.jpg)
Data come from Questions
![Page 3: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/3.jpg)
![Page 4: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/4.jpg)
Bad questions can mess up your findings
(in potentially embarrassing ways)
![Page 5: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/5.jpg)
Women: BMI by ‘Sexual Orientation’
2002 NSFG
HETEROSEXUAL HOMOSEXUAL BISEXUALNormal 49.7 30.5 48.5
Overweight 25.6 35.3 29.6
Obese 24.7 34.2 22.0
2006 NSFG
STRAIGHT OR HETEROSEXUAL
GAY OR LESBIAN OR HOMOSEXUAL BISEXUAL
Normal 40.7 38.2 36.5
Overweight 26.5 33.0 19.5Obese 32.8 28.8 44.0
![Page 6: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/6.jpg)
Data come from Questions
It is critical to understand how questions perform to understand what the data really are.
Why question evaluation is important…
![Page 7: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/7.jpg)
Goals for Question Evaluation:
How do the respondents understand the survey question?
Do respondents understand the survey question differently?
Does the question mean the same in all the languages that it is asked?
Does the question mean the same in all culture and socio-economic groups that it is asked?
In processing a question, do all respondents recall information and form an answer the same way?
![Page 8: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/8.jpg)
Goals for Question Evaluation:
To what extent are the data elicited from the question a true representation of the phenomena being studied?
In what ways is our picture distorted because the questions do not accurately capture the intended construct?
What important discovery are we not making because we are unaware that our picture is distorted?
![Page 9: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/9.jpg)
This Presentation
Describe QDRL and Question Evaluation at NCHS
Discuss question evaluation and cognitive interviewing
Examples along the way
![Page 10: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/10.jpg)
Question Design Research Lab
National Center for Health StatisticsOffice of Research and Methodology
FacilityLab (room with video cameras, microphones)Video data baseSoftware application for analyzing cognitive interviews
ActivitiesApplied Testing Projects (Cognitive Interviewing Studies)Research related to question evaluation methodology
![Page 11: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/11.jpg)
QDRL Guiding Principles1. The best question design is based on
question evaluation, not expert opinion
2. Question evaluation is science-based Empirical Evidence Transparent and Systematic analysis
3. Evaluation studies must be well documented, replicable, and made accessible
![Page 12: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/12.jpg)
Developments include:
Focus on analysisCombining various methods to improve
explanatory power of question performanceIntegration of qualitative and quantitative
methodologiesFocus on measuring comparability across
subgroups/disparity groups With an eye toward quantifying measurement
errorDevelopment of application suite:
Q-Bank, Q-Video, Q-Notes, Q-Notes Plus
![Page 13: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/13.jpg)
A good question is…1. relevant to the research agenda
and
2. relevant to each potential respondent’s experience and knowledge.
![Page 14: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/14.jpg)
What to Remember about Respondents
Do not know or understand the research question
Most likely, do not use scientific, abstract concepts
Survey puts them in the position of operating as informants
Reference aspects of their lives
![Page 15: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/15.jpg)
When this relationship is broken, error is introduced into the data.
False PositivesFalse NegativesAn entirely different phenomena is measured
than intended by the research agendaExample: Terrorism
![Page 16: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/16.jpg)
Question Response Process
RetrievalComprehension Judgment Response
Social FactorsSocial Factors
Social Factors
Social Factors
Social Factors Social Factors
Social Factors
![Page 17: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/17.jpg)
17
Expectations of a survey respondent:
An impromptu response is required
Responses must be categorizeable
![Page 18: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/18.jpg)
18
Expectations of a survey respondent:
An impromptu response is required
Responses must be categorizeable
Questions will be understood using formal language structure
![Page 19: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/19.jpg)
19
Expectations of a survey respondent:
An impromptu response is required
Responses must be categorizeable
Questions will be understood using formal language structure
The analytic purpose of questions will be intuited and responded to accordingly
![Page 20: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/20.jpg)
20
Expectations of a survey respondent:
An impromptu response is requiredResponses must be categorizeable Questions will be understood using
formal language structureThe analytic purpose of questions
will be intuited and responded to accordingly
Meaning to vague words, such as “mild,” “moderate,” “severe,” “extreme,” will be assumed through ordering
![Page 21: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/21.jpg)
21
Expectations of a survey respondent:
Inside the cultural frame of reference
Inside the questions’ system of knowledge
![Page 22: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/22.jpg)
22
Chronic Conditions
![Page 23: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/23.jpg)
23
Questions requiring an answer outside respondents’ knowledge
Chronic condition questionsMedicine questionsMedical test questions
![Page 24: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/24.jpg)
Cognitive InterviewsDesigned to understand how respondents
comprehend, retrieve, judge, respond to questions
Through this examination, can identifypotential response errors patterns of interpretation
Provide insight into social-cultural factors that impact the response process
![Page 25: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/25.jpg)
Cognitive Interviewing Study
QualitativeSmall sampleSample selection purposive Examines thought processes of
respondentHow does the question work?Does the question work as intended?
If not, how can it be “fixed”?
![Page 26: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/26.jpg)
Cognitive Interviews are Semi-structuredCore Question- interview is organized by
the questions that are being tested
Probe Questions- open-ended, spontaneous, not pre-scripted, based on the information that the respondent provides
![Page 27: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/27.jpg)
Probing for storyWhy did the respondent answer the
question the way that they did?
Does this story match with the intent of the question?
![Page 28: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/28.jpg)
Data from Cognitive TestingCollected from semi-structured protocolNarrative formatValidity tied to rich detailFindings are groundedInsight into question interpretationInsight into patterns of calculation
![Page 29: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/29.jpg)
Cognitive Interview FindingsProvide knowledge of question
performanceIllustrate what the question measures
Varied patterns of interpretationDimensions of response error
![Page 30: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/30.jpg)
Overall, during the past 4 weeks, how much difficulty did you have with thinking clearly and solving daily problems?
Respondent 5
Respondent 2
Respondent 6 Respondent 3
Respondent 4
Respondent 1
Alzheimer’s disease
Busy
Long term, medical problem
Specific experience- organizing tennants
Remembering detailed list
Fiscal functioning
![Page 31: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/31.jpg)
Cognitive Interviews
Designed to understand how respondents comprehend, retrieve, judge, respond to questions
Through this examination, can identifypotential response errors patterns of interpretation
Provide insight into social-cultural factors that impact the response process
Cognitive interview and question evaluation reports written and housed on Q-Bank
![Page 32: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/32.jpg)
Some Purposes of Cognitive Some Purposes of Cognitive InterviewingInterviewing1. Ensure the question is capturing intended
construct (validity) and document construct for data user
2. Hone construct so that it is meaningful and measureable
3. Assess comparability of translations4. Assess comparability across socio-cultural and
demographic subgroups (e.g. countries, SES status, age)
5. Determine/Confirm meaningful thresholds or cut-off points
6. Assess how similar questions work together (e.g. frequency and intensity questions)
![Page 33: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/33.jpg)
![Page 34: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/34.jpg)
“Case Study”
UNESCAP/Washington Group collaborationCambodia, Kazakhstan, Maldives, Mongolia, Cambodia, Kazakhstan, Maldives, Mongolia,
Philippines, Sri LankaPhilippines, Sri Lanka
To examine an extended set of disability questions with intent to develop measuresMobility, Vision, Hearing, Pain, Affect,
Fatigue, Cognition, Learning, Communication
Intended for global use
![Page 35: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/35.jpg)
This Study: Mixed Method This Study: Mixed Method ApproachApproach
A) Cognitive Interviewing: A) Cognitive Interviewing: Qualitative and Semi-structuredQualitative and Semi-structured143 interviews143 interviews
B) Field Interviewing: Quantitative B) Field Interviewing: Quantitative and Structuredand StructuredApprox. 1000 interviews per Approx. 1000 interviews per countrycountry
![Page 36: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/36.jpg)
Mixed Method approach to Question Evaluation and Measurement Construction
Cognitive Test: To understand the ways in which a question performs
Field Test: To understand the extent to which patterns
Objective is to identify:1. Patterns of interpretation2. Calculation processes3. Response error problems4. Interpretive patterns across
subgroups
Establish hypotheses for field test
![Page 37: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/37.jpg)
Cognitive Test: To understand the ways in which a question performs
Field Test: To understand the extent to which patterns exist
Objective is to determine:1. The extent of problematic
patterns2. The extent occurring in
particular subgroups
Test hypotheses from cognitive interviews
Mixed Method approach to Question Evaluation and Measurement Construction
![Page 38: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/38.jpg)
Example: Anxiety
UNESCAP/Washington Group Project
![Page 39: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/39.jpg)
Anxiety Questions
Frequency: How often do you feel worried, nervous or anxious? Daily, weekly, monthly, a few times a year, or never?
Intensity: Thinking about the last time you felt anxious, how would you describe the level of anxiety? Mild, moderate or severe?
![Page 40: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/40.jpg)
![Page 41: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/41.jpg)
![Page 42: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/42.jpg)
![Page 43: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/43.jpg)
Cognitive Interviewing Findings
Interpretations: Aspects of anxiety reported
Stress Workload, deadlines, performances Relationship/family problems Dread/concern about future
Economic security Health concerns
Clinical/diagnosed anxiety Excitement/Positive energy
![Page 44: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/44.jpg)
Additional Probe Questions
Please tell me which of the following statements, if any, describe your feelings.
1. My feelings are caused by the type and amount of work I do.
2. Sometimes the feelings can be so intense that my chest hurts and I have trouble breathing.
3. These are positive feelings that help me to accomplish goals and be productive.
4. The feelings sometimes interfere with my life, and I wish that I did not have them.
5. If I had more money or a better job, I would not have these feelings.
6. Everybody has these feelings; they are a part of life and are normal.
7. I have been told by a medical professional that I have anxiety.
In-Scope, Valid Interpretations
1. Chest Hurts: Sometimes the feelings can be so intense that my chest hurts and I have trouble breathing.
2. Interfere: The feelings sometimes interfere with my life, and I wish that I did not have them
3. Clinical: I have been told by a medical professional that I have anxiety.
Out-of-Scope, Not Valid Interpretations
1. Positive: These are positive feelings that help me to accomplish goals and be productive.
2. Normal: Everybody has these feelings; they are a part of life and are normal.
![Page 45: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/45.jpg)
Anxiety: Frequency
![Page 46: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/46.jpg)
Anxiety: Intensity
![Page 47: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/47.jpg)
Anxiety:Joint distribution of anxiety frequency and intensity
A few times a
year
Monthly Weekly Daily DK/REF
A little 1087 423 328 214 1
Closer to a little
35 25 27 12 0
In between 122 85 95 59 0
Closer to a lot 22 16 39 33 0
A lot 163 86 122 259 0
DK/REF 22 3 3 7 1
Note. Polychoric correlation = .42
![Page 48: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/48.jpg)
Additional Probe Questions
Please tell me which of the following statements, if any, describe your feelings.
1. My feelings are caused by the type and amount of work I do.
2. Sometimes the feelings can be so intense that my chest hurts and I have trouble breathing.
3. These are positive feelings that help me to accomplish goals and be productive.
4. The feelings sometimes interfere with my life, and I wish that I did not have them.
5. If I had more money or a better job, I would not have these feelings.
6. Everybody has these feelings; they are a part of life and are normal.
7. I have been told by a medical professional that I have anxiety.
In-Scope, Valid Interpretations
1. Chest Hurts: Sometimes the feelings can be so intense that my chest hurts and I have trouble breathing.
2. Interfere: The feelings sometimes interfere with my life, and I wish that I did not have them
3. Clinical: I have been told by a medical professional that I have anxiety.
Out-of-Scope, Not Valid Interpretations
1. Positive: These are positive feelings that help me to accomplish goals and be productive.
2. Normal: Everybody has these feelings; they are a part of life and are normal.
![Page 49: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/49.jpg)
Percentage of Reporting Anxiety Characteristics
![Page 50: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/50.jpg)
AnxietyPercent reporting various descriptions of anxiety by country
Kazakhstan
Cambodia
Sri Lank
aMaldiv
esMongoli
aPhilippine
s
Out-of-Scope, Not Valid Interpretations
Positive 50.3% 47.8% 12.6%
51.7% 82.5% 32.4%
Normal 81.5 71.3 75.4 86.7 85.7 81.2
In-Scope, Valid Interpretations
Chest hurts
21.4 72.3 30.9 37.0 50.6 20.3
Interfere
52.2 65.0 85.4 54.8 72.8 33.5
Clinical 11.8 16.8 3.0 28.4 18.6 11.5Note. Chi-square p <.05 for all rows in the table.
![Page 51: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/51.jpg)
“Validity” by countries
![Page 52: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/52.jpg)
Conclusion (A)
1. Question evaluation must be science-based Empirical Evidence Transparent and Systematic analysis With an audit trail, such as Q-Notes to
2. Evaluation studies must be well documented, replicable, and made accessible
3. Data users can improve their analyses with critical use of the data (i.e. questioning the validity of questions and the quality of data)
![Page 53: Basic Question Evaluation](https://reader033.vdocuments.mx/reader033/viewer/2022052903/55764e22d8b42aaa548b4645/html5/thumbnails/53.jpg)
Conclusion (B)A mixed-method approach for developing
cross-cultural measures:
Can capture interpretations of questionsProvides insight into the extent of those
interpretationsAllows for a comparability examination of
those interpretationsAllows for thoughtful, empirically based
decisions about the measures and the cut-off points