SuRGE-5’s User’s Guide:Survey Design within VR Performance Management
Darlene A. G. Groomes, Summit Reading Group Facilitator
Fundamental GroupKaren Carroll, Andrew Clemons, Elaine De Smedt, HarrietAnn Litwin, Matthew Markve, Janice McFall, Sukyeong Pi, Kellie Scott, Michael Shoemaker
Principal GroupDaniel Frye, Elisabeth Furber, Shelley Hendren, Russ Thelin, Ed Tos
7th Summit ConferenceLouisville, KentuckySeptember 2014
SuRGE: Summit Reading Groups for Excellence
• 5th Group: Fundamental and Principal Introductions• Learning Community of Summit Group• Professional Development and Training: Ten months and three combined
calls
• Represented General, Combined, and Blind VR Programs; TACE; Institutes of Higher Education; Disability Law & Advocacy
• Represented seven states across the nation
• Read Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method by Dillman, Smyth, and Christian (2009)
• Session PowerPoint, User Guide, Interactive Element, and Before/After Survey will be available at: http://vocational-rehab.com/summit-reading-groups/project-development/
• To provide VR agency leaders and staff with a resource • Improving performance through enhanced surveying methods
• Reviews survey design issues
• Explains different modes utilized to increase response rate
• Focus toward responders’ needs • Went beyond the book to include technology considerations and
satisfy accessibility issues for persons with disabilities.
• To receive data from surveying efforts that are complete, valuable, and reliable.
Vocational Rehabilitation Performance Management: A User’s Guide to Survey Design
Principal Group Comments
Internet, Mail, and Mixed-Mode Surveys
• When studying VR systems, surveys have often been used as a means to collect data.
• Such data has been used for various purposes • identify patterns• evaluate performance• identify areas for improvement• determine the future needs of individual VR
agencies and the overall VR system.
Survey Evolution
• Significant changes have occurred over the years with respect to how surveys are conducted and what modes or types of surveys are used.
• It is crucial for those conducting surveys to be aware of the various survey modalities and what they have to offer.
• Shift in recent years “Mixed-Mode” vs. “Single-Mode” surveys
“Mixed-Mode” vs. “Single-Mode” Surveys
• No longer assume a one-size-fits-all approach • Limitations in what a single mode survey may offer, such as
response rate• Response rate for telephone interviews has decreased
• decline of land lines• use of Caller ID to screen out phone calls• cell phone users not wanting minutes charged to their cell
phone bill
• Mixed-mode • Increased respondent rate• Allows individuals to respond to a particular type of survey
based on disability related needs
Specific Types of Survey Modes• In-Person Interview
The individual interviewed in person at their home or at another physical location.
• Mail SurveyThe respondent asked to send their response back in a self-addressed stamped envelope.
• Telephone SurveyCan be automated or conducted by live interviewers.
• Computer Assisted Personal Interviewing (CAPI)Responses entered into computer program on a laptop/other small computing device.
• Computer Assisted Self-Interviewing (CASI)The respondent enters own responses into computer/hand held device.
• Email Surveys and Internet SurveysSent directly to a respondent’s email address or hosted on a website.
• FaxDespite the decreased use of fax machines, for those who do not feel comfortable responding electronically.
Specific Types of Survey Modes continued
Provide Information about the Survey
• Surveyor clearly identifies who they are, the survey’s purpose, how the survey will be used, provides contact information, and offers assistance
• Build a positive relationship – rapport
Ask for Help or Advice
• Demonstrates the value of thoughts and opinions• Improves survey design and accessibility• Solicit input or advice from experienced surveyors inside and
outside of VR
Encouraging Participation
Thank Respondents
Appreciate respondents’ time and efforts:
• Thank participants throughout the survey process• Be considerate of participants’ time• Follow-up after completion
Support Group Mission and Values
• Clearly state the agency’s mission and values• Align the survey with mission and values• Inform respondents how the survey advances agency goals• Solicit respondents’ input
Make the Questionnaire Interesting
• Consider how wording and context affects interest in the survey• Explore formatting options which enhance aesthetic appeal• Accessibility considerations
Use Plain Language
• Avoid acronyms and agency-specific lingo• Be brief, objective, clear, and precise• Avoid ambiguous or redundant answer choices• Consider how respondents may misinterpret questions and
answer choices
Sequence Questions Logically
• Test survey to ensure logical flow• Group questions by content area – avoid revisiting topic areas• Use numerical or alphabetical order, when possible
Create an Incentive or Reward System
• Offer incentives whenever possible – respondents are offering time and effort
• If financial resources prohibit offering incentives, share how respondents’ efforts advance the organization’s mission
Make Responding Convenient
• Simple - opening an envelope, clicking a link• Keep surveys concise – respect time• Test survey across devices, platforms, and populations
Use Respectful and Comprehensible Language
• Be considerate of respondents’:• Education levels• Content-area knowledge
• Target 4th Grade reading level• Carefully review assumptions and cultural biases
Decreasing Barriers
Create a Short Questionnaire
• Avoid unnecessary, wordy, or overly detailed questions• Keep instructions clear and concise• Completion time: 5-10 minutes
Create a Simple Questionnaire
• Minimize the amount of answer choices per item• Provide simple, essential directions • Make answering easy (e.g. checkboxes)
Minimize Personal Information Requests
• Maintain confidentiality and anonymity• Clearly describe procedures for protecting personal information,
if necessary
Pilot Test All Surveys
• Surveys benefit from continuous improvement• Gather input from colleagues, experts, respondents, and
stakeholders before roll-out• Improve validity and utility through small group field tests
Visual Design“There are three responses to a piece of design – yes, no, and WOW! Wow is the one to aim for.” Milton Glaser
Design Elements• Words: need to convey unambiguous meaning
and to convey information• Numbers: sources of order and sequence• Symbols: used to convey meaning or
relationships that must be understandable to the respondent
• Graphics: shapes and visual additions that provide simple or complex information and meaning to the respondent
Design Properties
• Size: Affects impact and
meaning/IMPORTANCE• Font: Needs to be legible• Color and Shading: Determines how elements
stand out and relate to each other• Location: Impacts legibility and conveys how
elements are related to each other.
(or are not)
Grouping Principles• Symmetry: Regularity and balance
• Proximity: Nearness implies relatedness
• Similarity: visually similar will be grouped together
• Connectedness: connected will be grouped together
• Common Region: elements in a closed region will be grouped together
• Continuity: perception that one element leads to the next
• Closure: elements in a closed region will be perceived as grouped together
• Common Fate: elements perceived as moving in the same direction will be grouped together
Visual Guidelines/Recommendations
1. Use bold text for question and not bolded text for responses.
2. Use space to delineate groupings3. Make responses visually neutral 4. Emphasize important elements in the question
(e.g. underlining important phrases)5. Use a design element in a consistent manner
throughout the survey.6. Make visual and verbal messages consistent
Example of last pointAsks for the best (single) answer, but appears to have two sets of answers , calling for two responses.
Asks for the best (single) answer from a single list of responses.
vs
More Visual Guidelines/Recommendations
7. Put special instructions in the question rather than placing them in some other location.
8. Use an alternate visual cue to identify occasional or special instructions (e.g. italics).
9. Organize the information to be brief and to the point.
10. Choose line spacing, font and text size to ensure legibility of the text.
Choosing Your Words“Words mean more than what is set down on paper. It takes the human voice to infuse them with shades of deeper meaning.”
Maya Angelou, I Know Why the Caged Bird Sings
Two Kinds of Questions
• Open-ended questions:
specific answer choices are not provided
• Closed-ended questions:
specific answer choices are provided
1. To elicit a numerical response
2. Request for a list:
List your three favorite cerealsCereal 1Cereal 2Cereal 3
Use Open-Ended Questions:
Guidance: label the unit of response (e.g. days, hours, boxes, etc.)
Use Open-Ended Questions continued
3. When you want detailed, descriptive information.
Guideline 1: Use judiciously as narrative responses require individual coding.
Guideline 2: Allow sufficient space for the respondent to give a full answer.
Indicate if the text box will permit scrolling.
Use Close-Ended Questions1. In a neutral manner:
Rather than “do you agree?” state “do you agree or disagree?”
2. When using a Nominal Scale (asking to choose from one or more responses):
Guideline: when asking individuals to rank a series of items, you are more likely to get an accurate response if you individually pair each of the items.(e.g. A vs. B; B vs. C; A vs. C)
Nominal Scales continued
• Responses increase when the individual is asked to state yes or no to each item (rather that check “all that apply.”)
Indicate if you have visited each of these restaurants in the past 6 months:
YES NOPizza HutTaco BellMcDonald’sBurger King
Use Close-Ended Questions
3. Ordinal Scales measure gradation, either starting at “zero” (unipolar) or with a neutral option in the center “bipolar.”
Unipolar BipolarExcellent Strongly AgreeVery Good AgreeGood NeutralFair DisagreePoor Strongly DisagreeVery Poor
More about Bipolar Scales• Because respondents relate to the middle value as
“average,”Do this: Not this:Strongly Agree Strongly AgreeAgree AgreeNeutral NeutralDisagree DisagreeStrongly Disagree Strongly Disagree
Don’t KnowDon’t Know No AnswerNo Answer
Technology Considerations
Know your Audience• Match your survey type to those being surveyed
• Advantage of using mixed mode surveys when there may be limited access or comfort level with a web based or electronic format
• Consider how an electronic based survey may have an impact on confidentiality and response rates
Not all Internet Browsers are Alike
• Older computers may not be equipped to run up-to-date browsers
• Different operating systems at play (i.e., Windows vs. Apple)
• The impact of how surveys may be answered on different devices (i.e., Desktop systems, Iphones/Ipads, smartphones (Android) and tablets)
The Use of Internet-Based Survey Sites
• Survey Gizmohttp://www.surveygizmo.com/
• Survey Monkeyhttps://www.surveymonkey.com/
• Qualtricshttp://www.qualtrics.com/
Disability Accommodations
Back to Basics- Guidelines for Creating Surveys
• The Rehabilitation Act of 1973 as amended, Section 508 (U.S. Code, Title 29, Section 794d)
• Surveys given by a VR agency should allow for individuals with disabilities to have equal access through the use of reasonable accommodations and/or alternative formats.
Common Guidelines
• Use systems compatible with screen readers
• Allow for the ability to modify font size, color etc., provide text equivalents for every non-text element
• Create alternative tags for audio features to allow those with hearing difficulties to read what is being conveyed in the audio format
Common Guidelines continued
• Use systems compatible with speech recognition software
• Enable page shifting without the use of a mouse, and allow for keyboard alternatives for mouse commands
• Allowance for additional or needed response times to survey questions
Common Alternative Formats or Accommodations
• Surveys constructed in Braille format
• Telephone relay systems
• The use of readers or interpreters
• Examples of Section 508 Compliance in Internet surveys:
http://help.surveymonkey.com/articles/en_US/kb/Are-your-surveys-508-compliant-and-accessible
Mixed-Mode Surveys
• The use of several different modes in a survey procedure
• Examples of several different survey modes would include phone, on-line, mail and point of contact surveys
• The content of the survey changes minimally across modes so that data from the several modes used can be analyzed together
Potential Advantages to Using Mixed-Mode Surveys
1. Reduced Cost
2. Improved Time Lines
3. Reduced “Coverage” Error
Potential Advantages continued
4. Improved Response Rate and Reduction of Non-Response Error
5. Reduce Measurement Error
Type Motivation Limitations
1. Use of one mode to contact respondents and to encourage response by a different mode
Improve response rates Reduce coverage and nonresponse error
Increased implementation costs
2. Use a second mode to collect responses for the same respondents for specific questions within a survey
Reduce measurement error Reduce social desirability bias for sensitive questions
Increased design costs Increased nonresponse if respondent must respond by other mode at a later time
3. Use alternative modes for different respondents in the same survey period
Improve response rates Reduce coverage and nonresponse error Reduce survey costs
Increased design costs Measurement error from mode difference that may be confounded with difference among subgroups
4. Use a different mode to survey the same respondents in a later data collection
Different modes become available to survey respondents Reduce survey costs
Increased design costs Measurement error from mode differences that impact the ability to measure change over time
Customer Satisfaction Specifics and Delivery Methods
Customer Satisfaction Specifics• Discuss 10 guidelines for improving customer satisfaction
surveys by reducing errors in key areas:• Sampling methods• Nonresponse• Measurement/interpretation
Delivery Methods• Describe 3 survey methods which enhance accuracy
• In-person appeals with follow-up• Customer diaries• Group administration
Customer Satisfaction Specifics
• Creating valid customer satisfaction surveys can be a challenge; often the individuals designing the surveys are those being evaluated.
Sampling Methods1. Randomly sample populations instead of the entire
population, minimizing individuals being surveyed repeatedly/unnecessarily.
2. Develop procedures to ensure that onsite sampling is carefully executed and is not affected by personal preference.
Nonresponse3. Actively use follow-up reminders to reduce non-response
error.
4. Provide all respondents with similar amounts and types of encouragement.
Measurement/Interpretation5. Avoid encouraging higher ratings when delivering the survey
request.
6. Obtain responses when customers are best able to provide them.
Customer Satisfaction Specifics
Measurement/Interpretation7. Choose measurement devices which are credible to
respondents and surveyors.
8. Avoid choosing measurement devices primarily because of their potential to improve response rates.
9. Ensure scales are balanced and fully describe the measurement procedure when reporting survey results.
10. Evaluate the impact of using both aural and visual modes.
Customer Satisfaction Specifics
Delivery Methods
• Delivery methods enhance or undermine the validity of survey results
• How you deliver affects what you receive
• Selecting the appropriate delivery method reduces sampling, nonresponse, and measurement errors
• Delivery methods must match desired feedback
Delivery Methods
In-Person Appeals with Follow-Up Procedures• “Foot-in-the-door technique”• Increases validity by decreasing nonresponse error
Best practice implementation in stages: 1. Initial contact and in-person appeal2. Follow-up contacts with:
a) Reminders and/or thank you messagesb) The survey itself
• US National Park Service
Delivery Methods
Customer Diaries• Self-reporting behavior during a determined timeframe• Tracks behavior/use in addition to (or in lieu of)
satisfaction
• Procedural Timeline for Diary Surveys:a) Initial contact explains purpose, timelines, etc.b) Advance contact prior to mailing diary (e.g. 2-4 weeks)c) Mailing: diary, review of procedures, and gratitude/incentivesd) Reminders to customer prior, during, and after the diary periode) Final contact expresses gratitude, offers incentives, and/or
confirms receipt of completed diary
Delivery Methods
Group Administration• Ideal when timely feedback is necessary – when
customers’ memories must be fresh
• Guided group interviews can promote discussion - paper surveys allow anonymity
• Mixed-modes (e.g. paper, discussion, audio/video recordings) create robust data
• Skilled facilitation a key priority
For More InformationDarlene Groomes Summit Group Website248-370-4237 [email protected]
Fundamental GroupKaren Carroll, Andrew Clemons, Elaine De Smedt, HarrietAnn Litwin, Matthew Markve, Janice McFall, Sukyeong Pi, Kellie Scott, Michael Shoemaker
Principal GroupDaniel Frye, Elisabeth Furber, Shelley Hendren, Russ Thelin, Ed Tos
Thank you!