surge-5’s user’s guide: survey design within vr performance management darlene a. g. groomes,...

SuRGE-5’s User’s Guide:Survey Design within VR Performance Management

Darlene A. G. Groomes, Summit Reading Group Facilitator

Fundamental GroupKaren Carroll, Andrew Clemons, Elaine De Smedt, HarrietAnn Litwin, Matthew Markve, Janice McFall, Sukyeong Pi, Kellie Scott, Michael Shoemaker

Principal GroupDaniel Frye, Elisabeth Furber, Shelley Hendren, Russ Thelin, Ed Tos

7th Summit ConferenceLouisville, KentuckySeptember 2014

SuRGE: Summit Reading Groups for Excellence

• 5th Group: Fundamental and Principal Introductions• Learning Community of Summit Group• Professional Development and Training: Ten months and three combined

calls

• Represented General, Combined, and Blind VR Programs; TACE; Institutes of Higher Education; Disability Law & Advocacy

• Represented seven states across the nation

• Read Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method by Dillman, Smyth, and Christian (2009)

• Session PowerPoint, User Guide, Interactive Element, and Before/After Survey will be available at: http://vocational-rehab.com/summit-reading-groups/project-development/

• To provide VR agency leaders and staff with a resource • Improving performance through enhanced surveying methods

• Reviews survey design issues

• Explains different modes utilized to increase response rate

• Focus toward responders’ needs • Went beyond the book to include technology considerations and

satisfy accessibility issues for persons with disabilities.

• To receive data from surveying efforts that are complete, valuable, and reliable.

Vocational Rehabilitation Performance Management: A User’s Guide to Survey Design

Principal Group Comments

Internet, Mail, and Mixed-Mode Surveys

• When studying VR systems, surveys have often been used as a means to collect data.

• Such data has been used for various purposes • identify patterns• evaluate performance• identify areas for improvement• determine the future needs of individual VR

agencies and the overall VR system.

Survey Evolution

• Significant changes have occurred over the years with respect to how surveys are conducted and what modes or types of surveys are used.

• It is crucial for those conducting surveys to be aware of the various survey modalities and what they have to offer.

• Shift in recent years “Mixed-Mode” vs. “Single-Mode” surveys

“Mixed-Mode” vs. “Single-Mode” Surveys

• No longer assume a one-size-fits-all approach • Limitations in what a single mode survey may offer, such as

response rate• Response rate for telephone interviews has decreased

• decline of land lines• use of Caller ID to screen out phone calls• cell phone users not wanting minutes charged to their cell

phone bill

• Mixed-mode • Increased respondent rate• Allows individuals to respond to a particular type of survey

based on disability related needs

Specific Types of Survey Modes• In-Person Interview

The individual interviewed in person at their home or at another physical location.

• Mail SurveyThe respondent asked to send their response back in a self-addressed stamped envelope.

• Telephone SurveyCan be automated or conducted by live interviewers.

• Computer Assisted Personal Interviewing (CAPI)Responses entered into computer program on a laptop/other small computing device.

• Computer Assisted Self-Interviewing (CASI)The respondent enters own responses into computer/hand held device.

• Email Surveys and Internet SurveysSent directly to a respondent’s email address or hosted on a website.

• FaxDespite the decreased use of fax machines, for those who do not feel comfortable responding electronically.

Specific Types of Survey Modes continued

Provide Information about the Survey

• Surveyor clearly identifies who they are, the survey’s purpose, how the survey will be used, provides contact information, and offers assistance

• Build a positive relationship – rapport

Ask for Help or Advice

• Demonstrates the value of thoughts and opinions• Improves survey design and accessibility• Solicit input or advice from experienced surveyors inside and

outside of VR

Encouraging Participation

Thank Respondents

Appreciate respondents’ time and efforts:

• Thank participants throughout the survey process• Be considerate of participants’ time• Follow-up after completion

Support Group Mission and Values

• Clearly state the agency’s mission and values• Align the survey with mission and values• Inform respondents how the survey advances agency goals• Solicit respondents’ input

Make the Questionnaire Interesting

• Consider how wording and context affects interest in the survey• Explore formatting options which enhance aesthetic appeal• Accessibility considerations

Use Plain Language

• Avoid acronyms and agency-specific lingo• Be brief, objective, clear, and precise• Avoid ambiguous or redundant answer choices• Consider how respondents may misinterpret questions and

answer choices

Sequence Questions Logically

• Test survey to ensure logical flow• Group questions by content area – avoid revisiting topic areas• Use numerical or alphabetical order, when possible

Create an Incentive or Reward System

• Offer incentives whenever possible – respondents are offering time and effort

• If financial resources prohibit offering incentives, share how respondents’ efforts advance the organization’s mission

Make Responding Convenient

• Simple - opening an envelope, clicking a link• Keep surveys concise – respect time• Test survey across devices, platforms, and populations

Use Respectful and Comprehensible Language

• Be considerate of respondents’:• Education levels• Content-area knowledge

• Target 4th Grade reading level• Carefully review assumptions and cultural biases

Decreasing Barriers

Create a Short Questionnaire

• Avoid unnecessary, wordy, or overly detailed questions• Keep instructions clear and concise• Completion time: 5-10 minutes

Create a Simple Questionnaire

• Minimize the amount of answer choices per item• Provide simple, essential directions • Make answering easy (e.g. checkboxes)

Minimize Personal Information Requests

• Maintain confidentiality and anonymity• Clearly describe procedures for protecting personal information,

if necessary

Pilot Test All Surveys

• Surveys benefit from continuous improvement• Gather input from colleagues, experts, respondents, and

stakeholders before roll-out• Improve validity and utility through small group field tests

Visual Design“There are three responses to a piece of design – yes, no, and WOW! Wow is the one to aim for.” Milton Glaser

Design Elements• Words: need to convey unambiguous meaning

and to convey information• Numbers: sources of order and sequence• Symbols: used to convey meaning or

relationships that must be understandable to the respondent

• Graphics: shapes and visual additions that provide simple or complex information and meaning to the respondent

Design Properties

• Size: Affects impact and

meaning/IMPORTANCE• Font: Needs to be legible• Color and Shading: Determines how elements

stand out and relate to each other• Location: Impacts legibility and conveys how

elements are related to each other.

(or are not)

Grouping Principles• Symmetry: Regularity and balance

• Proximity: Nearness implies relatedness

• Similarity: visually similar will be grouped together

• Connectedness: connected will be grouped together

• Common Region: elements in a closed region will be grouped together

• Continuity: perception that one element leads to the next

• Closure: elements in a closed region will be perceived as grouped together

• Common Fate: elements perceived as moving in the same direction will be grouped together

Visual Guidelines/Recommendations

1. Use bold text for question and not bolded text for responses.

2. Use space to delineate groupings3. Make responses visually neutral 4. Emphasize important elements in the question

(e.g. underlining important phrases)5. Use a design element in a consistent manner

throughout the survey.6. Make visual and verbal messages consistent

Example of last pointAsks for the best (single) answer, but appears to have two sets of answers , calling for two responses.

Asks for the best (single) answer from a single list of responses.

vs

More Visual Guidelines/Recommendations

7. Put special instructions in the question rather than placing them in some other location.

8. Use an alternate visual cue to identify occasional or special instructions (e.g. italics).

9. Organize the information to be brief and to the point.

10. Choose line spacing, font and text size to ensure legibility of the text.

Choosing Your Words“Words mean more than what is set down on paper. It takes the human voice to infuse them with shades of deeper meaning.”

Maya Angelou, I Know Why the Caged Bird Sings

Two Kinds of Questions

• Open-ended questions:

specific answer choices are not provided

• Closed-ended questions:

specific answer choices are provided

1. To elicit a numerical response

2. Request for a list:

List your three favorite cerealsCereal 1Cereal 2Cereal 3

Use Open-Ended Questions:

Guidance: label the unit of response (e.g. days, hours, boxes, etc.)

Use Open-Ended Questions continued

3. When you want detailed, descriptive information.

Guideline 1: Use judiciously as narrative responses require individual coding.

Guideline 2: Allow sufficient space for the respondent to give a full answer.

Indicate if the text box will permit scrolling.

Use Close-Ended Questions1. In a neutral manner:

Rather than “do you agree?” state “do you agree or disagree?”

2. When using a Nominal Scale (asking to choose from one or more responses):

Guideline: when asking individuals to rank a series of items, you are more likely to get an accurate response if you individually pair each of the items.(e.g. A vs. B; B vs. C; A vs. C)

Nominal Scales continued

• Responses increase when the individual is asked to state yes or no to each item (rather that check “all that apply.”)

Indicate if you have visited each of these restaurants in the past 6 months:

YES NOPizza HutTaco BellMcDonald’sBurger King

Use Close-Ended Questions

3. Ordinal Scales measure gradation, either starting at “zero” (unipolar) or with a neutral option in the center “bipolar.”

Unipolar BipolarExcellent Strongly AgreeVery Good AgreeGood NeutralFair DisagreePoor Strongly DisagreeVery Poor

More about Bipolar Scales• Because respondents relate to the middle value as

“average,”Do this: Not this:Strongly Agree Strongly AgreeAgree AgreeNeutral NeutralDisagree DisagreeStrongly Disagree Strongly Disagree

Don’t KnowDon’t Know No AnswerNo Answer

Technology Considerations

Know your Audience• Match your survey type to those being surveyed

• Advantage of using mixed mode surveys when there may be limited access or comfort level with a web based or electronic format

• Consider how an electronic based survey may have an impact on confidentiality and response rates

Not all Internet Browsers are Alike

• Older computers may not be equipped to run up-to-date browsers

• Different operating systems at play (i.e., Windows vs. Apple)

• The impact of how surveys may be answered on different devices (i.e., Desktop systems, Iphones/Ipads, smartphones (Android) and tablets)

The Use of Internet-Based Survey Sites

• Survey Gizmohttp://www.surveygizmo.com/

• Survey Monkeyhttps://www.surveymonkey.com/

• Qualtricshttp://www.qualtrics.com/

Disability Accommodations

Back to Basics- Guidelines for Creating Surveys

• The Rehabilitation Act of 1973 as amended, Section 508 (U.S. Code, Title 29, Section 794d)

• Surveys given by a VR agency should allow for individuals with disabilities to have equal access through the use of reasonable accommodations and/or alternative formats.

Common Guidelines

• Use systems compatible with screen readers

• Allow for the ability to modify font size, color etc., provide text equivalents for every non-text element

• Create alternative tags for audio features to allow those with hearing difficulties to read what is being conveyed in the audio format

Common Guidelines continued

• Use systems compatible with speech recognition software

• Enable page shifting without the use of a mouse, and allow for keyboard alternatives for mouse commands

• Allowance for additional or needed response times to survey questions

Common Alternative Formats or Accommodations

• Surveys constructed in Braille format

• Telephone relay systems

• The use of readers or interpreters

• Examples of Section 508 Compliance in Internet surveys:

http://help.surveymonkey.com/articles/en_US/kb/Are-your-surveys-508-compliant-and-accessible

Mixed-Mode Surveys

• The use of several different modes in a survey procedure

• Examples of several different survey modes would include phone, on-line, mail and point of contact surveys

• The content of the survey changes minimally across modes so that data from the several modes used can be analyzed together

Potential Advantages to Using Mixed-Mode Surveys

1. Reduced Cost

2. Improved Time Lines

3. Reduced “Coverage” Error

Potential Advantages continued

4. Improved Response Rate and Reduction of Non-Response Error

5. Reduce Measurement Error

Type Motivation Limitations     

1. Use of one mode to contact respondents and to encourage response by a different mode

Improve response rates Reduce coverage and nonresponse error 

Increased implementation costs

2. Use a second mode to collect responses for the same respondents for specific questions within a survey

Reduce measurement error Reduce social desirability bias for sensitive questions

Increased design costs Increased nonresponse if respondent must respond by other mode at a later time

3. Use alternative modes for different respondents in the same survey period

Improve response rates Reduce coverage and nonresponse error Reduce survey costs

Increased design costs Measurement error from mode difference that may be confounded with difference among subgroups 

4. Use a different mode to survey the same respondents in a later data collection

Different modes become available to survey respondents Reduce survey costs

Increased design costs Measurement error from mode differences that impact the ability to measure change over time

Customer Satisfaction Specifics and Delivery Methods

Customer Satisfaction Specifics• Discuss 10 guidelines for improving customer satisfaction

surveys by reducing errors in key areas:• Sampling methods• Nonresponse• Measurement/interpretation

Delivery Methods• Describe 3 survey methods which enhance accuracy

• In-person appeals with follow-up• Customer diaries• Group administration

Customer Satisfaction Specifics

• Creating valid customer satisfaction surveys can be a challenge; often the individuals designing the surveys are those being evaluated.

Sampling Methods1. Randomly sample populations instead of the entire

population, minimizing individuals being surveyed repeatedly/unnecessarily.

2. Develop procedures to ensure that onsite sampling is carefully executed and is not affected by personal preference.

Nonresponse3. Actively use follow-up reminders to reduce non-response

error.

4. Provide all respondents with similar amounts and types of encouragement.

Measurement/Interpretation5. Avoid encouraging higher ratings when delivering the survey

request.

6. Obtain responses when customers are best able to provide them.

Customer Satisfaction Specifics

Measurement/Interpretation7. Choose measurement devices which are credible to

respondents and surveyors.

8. Avoid choosing measurement devices primarily because of their potential to improve response rates.

9. Ensure scales are balanced and fully describe the measurement procedure when reporting survey results.

10. Evaluate the impact of using both aural and visual modes.

Customer Satisfaction Specifics

Delivery Methods

• Delivery methods enhance or undermine the validity of survey results

• How you deliver affects what you receive

• Selecting the appropriate delivery method reduces sampling, nonresponse, and measurement errors

• Delivery methods must match desired feedback

Delivery Methods

In-Person Appeals with Follow-Up Procedures• “Foot-in-the-door technique”• Increases validity by decreasing nonresponse error

Best practice implementation in stages: 1. Initial contact and in-person appeal2. Follow-up contacts with:

a) Reminders and/or thank you messagesb) The survey itself

• US National Park Service

Delivery Methods

Customer Diaries• Self-reporting behavior during a determined timeframe• Tracks behavior/use in addition to (or in lieu of)

satisfaction

• Procedural Timeline for Diary Surveys:a) Initial contact explains purpose, timelines, etc.b) Advance contact prior to mailing diary (e.g. 2-4 weeks)c) Mailing: diary, review of procedures, and gratitude/incentivesd) Reminders to customer prior, during, and after the diary periode) Final contact expresses gratitude, offers incentives, and/or

confirms receipt of completed diary

Delivery Methods

Group Administration• Ideal when timely feedback is necessary – when

customers’ memories must be fresh

• Guided group interviews can promote discussion - paper surveys allow anonymity

• Mixed-modes (e.g. paper, discussion, audio/video recordings) create robust data

• Skilled facilitation a key priority

For More InformationDarlene Groomes Summit Group Website248-370-4237 www.vocational-rehab.comgroomes@oakland.edu

Fundamental GroupKaren Carroll, Andrew Clemons, Elaine De Smedt, HarrietAnn Litwin, Matthew Markve, Janice McFall, Sukyeong Pi, Kellie Scott, Michael Shoemaker

Principal GroupDaniel Frye, Elisabeth Furber, Shelley Hendren, Russ Thelin, Ed Tos

Thank you!