CAUCE DIAL AND DISCUSSTUESDAY, 23 FEBRUARY 2010
STANLEY VARNHAGENU n i v e r s i t y o f A l b e r t a
KAREN HAYWARDU n i v e r s i t y o f S a s k a t c h e w a n
Using Surveys to Maximize Strategic Planning
Dial and Discuss: Using Surveys
2
Overview
Surveys and the Planning ProcessUnderstanding SurveysDelivery MethodsClosing Tips
23 February 2010
Dial and Discuss: Using Surveys
3
Surveys and the Planning Process
23 February 2010
Dial and Discuss: Using Surveys
4
Surveys and Planning
23 February 2010
Are surveys needed/useful for planning or do they just get in the way (slow things down)?
Depends on the timing of the planning Useful for evidence-based decisions Surveys can be an efficient means of collecting data Properly done, they can be a valuable tool that
increases the chances of success
Dial and Discuss: Using Surveys
5
Surveys are useful when…
23 February 2010
information is needed to examine population characteristics
a larger response pool is neededexamining subgroup data is important (e.g.,
gender or age differentials)breadth of response is more important than
depth
Dial and Discuss: Using Surveys
6
Surveys & Evaluations
23 February 2010
Surveys are effective tools with most types of evaluations including: Needs assessments
Collecting information about the target audience Formative evaluations (midterm)
Collecting information that helps make adjustments & improvements
Summative evaluations Collecting information that helps determine if goals and
objectives are being met (merit, value, & worth) Maintenance evaluations
Collecting information a period of time after whatis being evaluated is complete, to see if the objectives are still being met
Dial and Discuss: Using Surveys
7
Surveys and Strategic Planning
23 February 2010
Surveys can help us:Better understand potential
audiences/users/students (consider different segments)
Collect baseline data that can be compared with subsequent survey data
Test assumptions or preliminary products (trial balloons)
Examine existing and potential markets
Dial and Discuss: Using Surveys
8
Understanding Surveys…
23 February 2010
Dial and Discuss: Using Surveys
9
23 February 2010
What Surveys Can Offer
Collect suggestions to improveCatch problemsTrack trendsGet anonymous feedbackGet buy-in
Gain understandingIncrease awareness
2 way processEasy and effective way to collect dataWith an appropriate sample and response
rate quantitative data can help generalize to the population
Dial and Discuss: Using Surveys
10
23 February 2010
Strengths of Surveys
Easy to createEasy to analyzeRelatively inexpensiveCommon/respondent familiarityHandle both quantitative and qualitative
dataCan be relatively time efficient to takeReal time resultsInstant Feedback
Dial and Discuss: Using Surveys
11
Additional Reasons to Use Surveys
23 February 2010
Can be cost savings, to allow feedback before considerable time and resources spent Storyboards/scripts
Allow feedback while modification is still possible Helpful to the developer and the user
Determine if goals and objectives have been met
Dial and Discuss: Using Surveys
12
23 February 2010
Weaknesses of Surveys
Lack of follow-upNot best suited for collecting qualitative
dataLimited flexibilitySampling problems not always obviousCan miss key results
Results are dependent on the questions/options that are presented
Garbage In Garbage Out (GIGO)
Dial and Discuss: Using Surveys
13
Delivery Methods
23 February 2010
Dial and Discuss: Using Surveys
14
Survey Delivery Methods
23 February 2010
Surveys can be conducted using different delivery methods These methods have different strengths and
weaknesses that should be considered The three different delivery methods considered here
are: Online Paper-based Telephone
Currently Online Surveys are probably themost popular method
Dial and Discuss: Using Surveys
15
23 February 2010
Online
Advantages: Minimal distribution
costs Automatic data entry Automatic
compilation possible Branching possible Potentially quickly
modified or updated Real time data
collection potential Can use
multimedia
Disadvantages: SPAM Filters/Broken
Links Duplication a concern User competency
dependent (submit) More complicated Anonymity a concern Potential user cost (Long
distance/ISP) Access
Dial and Discuss: Using Surveys
16
Other Online Considerations
23 February 2010
Response rateLengthItem types
Forced choicesMaking changesNavigation
Going to previous pages
Submitting surveyAccommodating
schedules
Different browsers Colors Size
Exactly who is responding
Location of the survey server
Online survey fatigue
Dial and Discuss: Using Surveys
17
23 February 2010
Paper and Pencil
Advantages: Simple (for
respondents and evaluator) No special training
required Easy to go back and
change responses Can have a higher
response rate
Disadvantages: Distribution
Can be slow Branching Following
instructions Usually required
manual data entry Reading writing
Qualitative responses Costs
Dial and Discuss: Using Surveys
18
23 February 2010
Phone
Advantages Can sometimes
provides higher response rates
Ability to branch Limited ability to
follow up response Allows assessment of
response quality Longer surveys
possible
Disadvantages Cost
Interviewer Long distance Persistence
More obtrusive Can be perceived
anonymity problems Selective responses Caller ID & Voicemail
Telemarketers
Dial and Discuss: Using Surveys
19
General Survey Considerations
23 February 2010
Use (needs assessment, formative, or summative)
Data entry methodNumber of
response options 5, 7, 99, or unlimited Even or odd
Effective scale
Anchors Continuum versus
descriptorsStatement
questionsNormative
referencedValidityReliability
Dial and Discuss: Using Surveys
20
Other Considerations
23 February 2010
Completion Time Written versus
telephone Class versus own
timeMixed methodsTriangulationPiloting
IncentivesRoutingFundingBiasTerminology
AcronymsReading ability
Dial and Discuss: Using Surveys
21
23 February 2010
Other Considerations (cont’d)
Anonymity Aggregating data Removing identifiers Looking at trends across surveys
Differential responses Only the “keeners” complete or vice-versa Some other important characteristic determines
who respondsSamplingSocial desirabilityTime survey is available
Dial and Discuss: Using Surveys
22
Bias
23 February 2010
Can affect results, but is often not obviousQuestion wording
Leading questionsOptions available
Missing optionsQuestion order
Specific to general Carry over
Option orderGeneral survey expectations
Dial and Discuss: Using Surveys
23
Creating Surveys… Examples
23 February 2010
Dial and Discuss: Using Surveys 23 February 2010
25
Item Types
Likert - Interval data how many options even or odd additional options direction anchors
I feel very comfortable taking online distance courses.1 2 3 4 5Strongly Disagree Neutral Agree Stronglydisagree agree
Dial and Discuss: Using Surveys
26
23 February 2010
End-of-course Evaluations
Uses Likert ItemsFive optionsStandardizedNormative
Effective scaleSocial desirabilityHow would we know when assumptions have
been violated?Intuitive meaning
Dial and Discuss: Using Surveys 23 February 2010
27
Item Types (continued)
Modified Likert Not agree/disagree items Continuum (options) changes Can use fewer anchors
2, 3, or 5
How comfortable do you feel online distance courses?1 2 3 4 5Not at all Somewhat Verycomfortable comfortable comfortable
Dial and Discuss: Using Surveys
28
23 February 2010
Item Types (continued)
Multiple choice Nominal data
Reporting results Picking the correct
choice Instructions
Select best Select all that apply
Missing options
Where do you plan to complete most of your online coursework?
a)Homeb)Workc) Computer labsd)Not suree)Other (please specify):
Dial and Discuss: Using Surveys
29
23 February 2010
Item Types (continued)
Rankings Following instructions Can force distinctions Can be harder to interpret results
Fill-in-the-blanks Can be time consuming to code Generalization
Open ended Use open-ended questions carefully
Exploratory versus confirmatory Time Potentially rich
Dial and Discuss: Using Surveys
30
Closing Tips
23 February 2010
Dial and Discuss: Using Surveys
31
23 February 2010
Survey Writing Tips
Demographic information Focus on what information will be useful for
analysis Recognize many consider the information given
to be sensitive Caution about subject identification
Trade-offs Don’t make responses too easy
Dial and Discuss: Using Surveys
32
23 February 2010
Survey Tips (cont’d)
Instructions Clarify purpose/reasons Anonymity Voluntary Withdrawal Ethics
Implied Consent Dissemination
When and where results will be found
Dial and Discuss: Using Surveys
33
23 February 2010
Survey Pitfalls
Want to get the largest proportion possible to respond Reduced the chance of self-selection Increases likelihood of accurately generalizing
resultsIncreasingly difficult to get high response
rates Frequency being surveyed
Survey fatigue Perceived need/relevance
Dial and Discuss: Using Surveys
34
Tips to Increase Response Rates
23 February 2010
Well written cover letter from meaningful individual Explaining survey purpose and relevance
Honorarium – given with survey Draw (e.g., gift certificate)
Keep survey as short as possibleProvide reasonable deadlineReminders
Dial and Discuss: Using Surveys
35
23 February 2010
Reference
Dillman, D. A., Smyth, J. D., & Christian, L. M. (2009) Internet, Mail, and Mixed-Mode
Surveys: The Tailored Design Method (Third Edition). New York: John Wiley & Son.
Dial and Discuss: Using Surveys
36
Questions?
23 February 2010
Dial and Discuss: Using Surveys
37
23 February 2010
Contacting Us
Stanley Varnhagen University of Alberta phone: (780) 492-3641 email: [email protected]
Karen Hayward University of Saskatchewan phone: (306) 966-7384 email: [email protected]