designing & implementing surveys anthony m. paolo, ph.d. office of medical education &...
TRANSCRIPT
Designing & Implementing Surveys
Anthony M. Paolo, Ph.D.Office of Medical Education &
Teaching and Learning Technologies
2
Overview
• What is a survey?
• Designing a survey
• KUMC survey tools
3
What is a Survey?Method of collecting information.
Good at gathering “how” and/or “what” people think/feel about issues.
Not good at getting at “why” people think the way they do.
4
Stages of Implementing Surveys
1. Defining objectives
2. Method of data collection
3. Selecting a sample
4. Writing questions
5. Designing the survey
6. Cleaning and verifying data
7. Analyzing data & reporting results
5
Define ObjectivesPrecisely defined objectives are the basis for the Population/sample, Questions, Analysis, and Reporting.
Good objectives have 5 elementsWho will do how much (how well) of what by when
“Determine the quality of education at KU”
“Student satisfaction with the course syllabus”
6
Stages of Implementing Surveys
1. Defining objectives
2. Method of data collection
3. Selecting a sample
4. Writing questions
5. Designing the survey
6. Cleaning and verifying data
7. Analyzing data & reporting results
7
Method of Data Collection1
• Hand out survey
• Self-administered mailed survey
• Self-administered web-based survey
• Face-to-face interview
• Telephone interview
• Method used determined by objectives, available time, resources, and sample.
8
Method of Data Collection2
Web-based / e-mail surveys • Advantages
– Very cost effective– Easy to develop, collect, analyze and respond to– Reduce bias since everyone gets the same item
presentation– Less intrusive then phone or face-to-face interviews– Get more and longer written comments then mailed
surveys– Becoming more popular
9
Method of Data Collection3
Web-based / e-mail surveys
• Disadvantages– Tend to have lower response rates then
mailed surveys– Cannot probe for deeper responses– Assume person sent the survey to is the one
that completed it. – People that have e-mail/web access may not
be demographically representative of general population
10
Stages of Implementing Surveys
1. Defining objectives
2. Method of data collection
3. Selecting a sample
4. Writing questions
5. Designing the survey
6. Cleaning and verifying data
7. Analyzing data & reporting results
11
Selecting a Sample1
• Define Population
• Survey entire population or just a sample?– Depends on resources, time, & size of
population
• The best samples are randomly selected and representative of the population on important characteristics.
12
Selecting a Sample2
Sample Types1. Probability
• Systematic random• Stratified random
2. Non-probability• Convenience• Snowballing
– Main disadvantage: Difficult to generalize results
13
Selecting a Sample3
How big of a sample?
• The bigger the sample, the more reliable the results.
• Depends on your objectives– Comparing groups?
• Can do a Power Analysis
• General Guideline– Have at least 30 in each subgroup, 50 to 100
is better.
14
Stages of Implementing Surveys
1. Defining objectives
2. Method of data collection
3. Selecting a sample
4. Writing questions
5. Designing the survey
6. Cleaning and verifying data
7. Analyzing data & reporting results
15
Writing Questions1
• Based on the objectives
• When in doubt, write fewer questions rather than more.
• Questions should be short & clear.
16
Writing Questions2
• Ask for an answer on only one dimension– Do you like the texture & flavor of the food?
• Accommodate all possible answers– What brand of computer to you own?
A. IBM PCB. Apple
• Have mutually exclusive options– Where did you grow up?
A. CountryB. FarmC. City
17
Writing Questions3
• Do not assume a certain state of affairs– Are you satisfied with your car insurance?
A. Yes B. No
• Avoid abbreviations and double negatives.
• Avoid emotionally loaded or vaguely defined terms.
– Crisis, failure, superb– Usually, frequently, rarely
Item Writing – Vague Terms
18
19
Writing Questions4
• Write your items to at most an 8th grade reading level.
• Do not imply a desired answer– Don’t you think congress spends too much
money?
• Do not ask respondents to order or rank more than 5 items. If must rank, better to have subjects rate each object separately and you rank them.
20
Writing Questions5
Response Scales
• How many options?– 5 to 7 is most adequate
• Include Middle/Neutral Option?
• Include Don’t Know/NA?
21
Writing Questions6
Response Scales
• Positive on left or right?
22
Stages of Implementing Surveys
1. Defining objectives
2. Method of data collection
3. Selecting a sample
4. Writing questions
5. Designing the survey
6. Cleaning and verifying data
7. Analyzing data & reporting results
23
Designing the Survey1
Make respondent’s task as easy and simple as possible.
Items on survey should be grouped in coherent sections.
Questions that use the same response format should appear together.
Include open-ended questions.
24
Designing the Survey2• Provide a cover letter (no more than 1 page)
– Describe purpose of survey and it’s importance to your organization and the respondent
– Mention how long it will take to complete the survey– Mention incentives, if applicable– Encourage prompt response and mention when survey closes– Describe any confidentiality/anonymity issues– Give information of contact person– Thank respondent for participating
• Provide simple and clear directions• Pre-notification letters increase response rate• Reminders to non-responders that include a copy of the
survey increase response rate.• Longer surveys tend to get lower response rates
Keep It Short!!!
25
Designing the Survey3
• Pilot Testing– Should be done with sample similar in
characteristics to the population you are surveying.
– Use small groups and be present when they complete the survey.
– “Talk Out Loud” method
26
Designing the Survey4
Reliability - A reliable survey will provide a consistent measure of important characteristics.
Validity – Are you measuring what you want to measure?
27
Stages of Implementing Surveys
1. Defining objectives
2. Method of data collection
3. Selecting a sample
4. Writing questions
5. Designing the survey
6. Cleaning and verifying data
7. Analyzing data & reporting results
28
Cleaning & Verifying Data• Check for:
– Missing data– Out of range data– Duplicates – Multiple response items– Inconsistent responses– Don’t Know, NA
29
Stages of Implementing Surveys
1. Defining objectives
2. Method of data collection
3. Selecting a sample
4. Writing questions
5. Designing the survey
6. Cleaning and verifying data
7. Analyzing data & reporting results
30
Analyzing & Reporting Results1
• Response rate The higher the better 80% or better - no worries < 50% - worry
• Representativeness – Compare respondents and non-respondents
on important characteristics.
31
Analyzing & Reporting Results2
How results are reported depends on the type of questions and the audience.
Statistical Analyses
Sampling error [(P(1-P)) / n] 1/2
Margin of Error – Sampling Error
n 95% CI
10 31%
30 18%
50 14%
100 10%
200 7%
300 6%
500 4%
1000 3%
1500 3%
2000 2%
32
33
Analyzing & Reporting Results3
• Statistical Significance p-value < .05Statistical significance means that the finding is rare, probably not due to chance or error.
It does not mean the finding is important or even meaningful
34
Analyzing & Reporting Results4
Suppose you find:• Males have a GPA of 3.50 (SD = .30)
• Females have a GPA of 3.54 (SD = .30)
• Is the difference of .04 meaningful?
• Is the difference statistically significant?
It is not meaningful (Effect size = .13), but is statistically significant (p < .05) with 500 per group
35
Software Available for Surveys• Angel• Vovici • SurveyMonkey
36
References• Fink A, The survey kit, Sage • Fink A, How to conduct surveys, Sage• Fowler FJ, Improving survey questions, Sage• Schuman H, & Presser, S. Questions & answers
in attitude surveys, Sage• Henerson ME, Morris LL, Fitz-Gibbon CT, How
to measure attitudes, Sage• Dillman DA, Mail & internet surveys, John Wiley
& Sons
37
Questions / Comments?