using surveys to improve your library - part 2
TRANSCRIPT
Using surveys to improve your library
Emily Daly & Joyce Chapman Duke University Libraries
ALA eLearning Workshop, February 2017
IntroductionsEmily Daly, Head of Assessment
& User ExperienceJoyce Chapman, Assessment
Analyst & Consultant
Agenda
• Survey validation and piloting• Basic survey structure • Writing actionable survey questions• Survey tools (briefly!)• Acting on survey data• Key takeaways
Time to hear from you!
Which methods do you use to distribute surveys in your library?
SURVEY VALIDATION AND PILOTING
What is survey validation?
• The process of assessing the survey questions for their dependability
• Have two parties review the survey if possible:1. People familiar with
the topic2. “Expert” in survey
question design
Who are these “experts”?Public libraries might get help from:– A staff person at the State Library– City/county government staff working in
assessment/planning
College/university library might get help from:– Assessment/UX staff in the library– Assessment office on campus
Validating the survey• Start with a simple text document• Have validators go through the survey and
make notes– Do all questions and answer choices make sense,
are they unbiased, etc.? – Will the resulting data help you answer your
questions? Is all topical content accurate?• Make changes based on validation!
What is survey piloting?
• Select a small subset of your target population to take your survey
• Even one pilot tester is better than none!
• Try to get a range of different people who represent your target group
Pilot testing your survey
• Pilot one by one• Pilot using your final
tool (digital or paper)• Observe, ask them to
think out loud, or to write notes
• Time respondents• Revise your survey and
re-test if desired
Questions or comments?
STRUCTURE OF THE SURVEY
Form follows function• What are you most interested in learning?• Who will analyze the data? How? • Who will see the results? • Who is your target response group? • How much time do they have?• How invested are your respondents?
The introductory statement
Some examples“The Duplin County Public Library is interested in learning more about what community members think of services provided by the library.”
Some examples“The Duplin County Public Library is interested in learning more about what community members think of services provided by the library.”
“The purpose of this study is to identify areas undergraduate students feel should be addressed in order to maintain an effective academic library.”
More examples“The library is particularly interested in the opinions of patrons who live in the western part of the county. You have been selected at random from community residents in western Greene County.”
More examples“The library is particularly interested in the opinions of patrons who live in the western part of the county. You have been selected at random from community residents in western Greene County.”“Your opinions are very important to us. There are no right or wrong answers. Your responses will be treated confidentially. Survey results will in no way be traceable to individual respondents.”
Question sequence
Strategies for good sequence• Start with well formatted, engaging questions• Don’t lead with questions on sensitive topics • Place questions you care about most in the
first half • Avoid repetitive, consecutive questions that
lead to reflexive responses • Consider ending with demographics questions
Status updates
Image: www.clustershot.com/garysimmons/photo649731
Image: https://flic.kr/p/coyARS
Closed responsesAdvantages• Uniform response set facilitates comparison,
choices clarify meaning of question, reminder of alternatives, pre-establishment for sensitive questions, increased response rate and speed
Closed responsesAdvantages• Uniform response set facilitates comparison,
choices clarify meaning of question, reminder of alternatives, pre-establishment for sensitive questions, increased response rate and speed
Disadvantages• Random selection, “closest representation”
issue, loss of distinction
Open responsesAdvantages• Allow for deep explanations, only way to get
responses you wouldn’t otherwise know
Open responsesAdvantages• Allow for deep explanations, only way to get
responses you wouldn’t otherwise knowDisadvantages• Requires communication skills from
respondents, more time consuming for respondent and analysis
Compromise Would you recommend this search results screen to a friend or colleague? Yes
Maybe No
Compromise Would you recommend this search results screen to a friend or colleague? Yes
Maybe No
Why, or why not?
Last call for feedback!
Last call for feedback!
Any additional comments about the Natrona County Public Library?
Last call for feedback!
Any additional comments about the Natrona County Public Library?
What else would you like to tell us about your experience using this page?
Last call for feedback!
Any additional comments about the Natrona County Public Library?
What else would you like to tell us about your experience using this page?
Is there anything else you’d like to share?
Closing your survey
• Demographics questions for data you can’t get elsewhere
• Call for volunteers• Clear SUBMIT button• Thank respondents for participating!• Web-based: Redirect to a related page or the
library homepage
Invitations to participate• “Would you be willing to participate in future
discussions or focus groups about the library? If so, please provide your contact information below.”
Invitations to participate• “Would you be willing to participate in future
discussions or focus groups about the library? If so, please provide your contact information below.”
• “Would you be interested in joining a community of makers to present their work in the Maker Space?” – Yes, I'm interested in being contacted about this
opportunity – No, I'm not interested in being contacted about this
opportunity
Questions or comments?
WRITING UNBIASED, ACTIONABLE, EFFECTIVE QUESTIONS
Avoid jargon and colloquialisms Think of your audience. Wording should be simple. Avoid technical words.
X "How many times last week did you use the library's Internet-enabled public access computers?"
Avoid jargon and colloquialisms Think of your audience. Wording should be simple. Avoid technical words.
X "How many times last week did you use the library's Internet-enabled public access computers?"
O "How many times last week did you use the library's computers [to access the Internet]?"
Abbreviations and acronymsAssume your audience does not know any of these.
X “CCPL is interested in replacing its OPAC."
X “How frequently do you use our ILL services?”
Abbreviations and acronymsAssume your audience does not know any of these.
X “CCPL is interested in replacing its OPAC."O “Cleveland County Public Library is interested in replacing its
online catalog.”X “How frequently do you use our ILL services?”
O “How frequently do you use our Interlibrary Loan services? [This service allows us to request material from another library
for you if we do not have it here].”
Avoid ambiguityLook at the survey from every angle: are there ways that someone could interpret a question to have two meanings?
X "What is your income?"
X "How many people are there in your household?"
Avoid ambiguityLook at the survey from every angle: are there ways that someone could interpret a question to have two meanings?
X "What is your income?"O "What is your income before taxes? Include salary as well as
other sources of income."
X "How many people are there in your household?"O "Including yourself, how many people are there in your
household?"
Confusing phrasingThe respondent should not have to spend time re-reading/interpreting the question.
X "Does it seem likely or does it seem unlikely to you that you would use a Maker Space if the library had one?"
It seems likely __ It seems unlikely __ I’m not sure __
Confusing phrasingThe respondent should not have to spend time re-reading/interpreting the question.
X "Does it seem likely or does it seem unlikely to you that you would use a Maker Space if the library had one?"
It seems likely __ It seems unlikely __ I’m not sure __
O "If the library had a Maker Space, would you use it?"Yes__ No__ Unsure__
Avoid double barreled questionsA question that introduces two or more issues with the expectation of a single response
X "Is our staff friendly and professional?"
Avoid double barreled questionsA question that introduces two or more issues with the expectation of a single response
X "Is our staff friendly and professional?"
O Question1: "Is our staff friendly?"O Question 2: "Is our staff professional?"
Avoid non-specific questionsDo not leave questions open to a wide range of interpretations.
X “How do you feel about public transportation?”
Avoid non-specific questionsDo not leave questions open to a wide range of interpretations.
X “How do you feel about public transportation?”
O “How do you feel about the DATA bus system in Durham County, North Carolina?”
Manipulative informationCertain questions require some background. Be careful that explanatory statements do not unduly influence responses.
X The county government spends approximately $10 per resident on landscaping public areas. Do you believe that the county government is adequately allocating funds for our library by designating only $1.15
per resident?"
Manipulative informationCertain questions require some background. Be careful that explanatory statements do not unduly influence responses.
X The county government spends approximately $10 per resident on landscaping public areas. Do you believe that the county government is adequately allocating funds for our library by designating only $1.15
per resident?" O "Do you believe that the county government is adequately allocating
funds for our library by designating $1.15 per resident?"
Manipulative informationWe often are interested to know how knowledge of the difference in spending might affect responses, first ask straightforward, then with additional info.
O "Do you believe that the county government is adequately allocating funds for our library by designating $1.15 per
resident?" O "If you were to learn that the county government spends approximately $10 per resident on landscaping public areas,
would that change your opinion about the adequacy of allocating $1.15 per resident to the library?"
Order of response optionsOften there is a logical, inherent order. If order is irrelevant, list choices alphabetically so respondents don't assume answers at the top are more important to the interviewer, or have software randomize them.
X Group study rooms Digital media lab Laptop lending E-books Printing/copying
Order of response optionsOften there is a logical, inherent order. If order is irrelevant, list choices alphabetically so respondents don't assume answers at the top are more important to the interviewer, or have software randomize them.
X Group study rooms O Digital media lab Digital media lab E-books Laptop lending Group study rooms E-books Laptop lending Printing/copying Printing/copying
Interval categoriesDo not allow to overlap. Provide an unbounded final category if appropriate.
X Age 0-10 Age 10-15 Age 15-20 Age 20-50 Age 50-75
Interval categoriesDo not allow to overlap. Provide an unbounded final category if appropriate.
X Age 0-10 O Age 0-9 Age 10-15 Age 10-19 Age 15-20 Age 20-29 Age 20-50 Age 30-39 Age 50-75 Age 40-49
Age 50+
Multiple response clarificationSometimes we allow respondents to choose only one option and sometimes we let them choose multiple. Be very clear that you are allowing multiple! Otherwise results are unclear.
X For which of the following reasons do you use the library?X__ Y__ Z__
Multiple response clarificationSometimes we allow respondents to choose only one option and sometimes we let them choose multiple. Be very clear that you are allowing multiple! Otherwise results are unclear.
X For which of the following reasons do you use the library?X__ Y__ Z__
O For which of the following reasons do you use the library? Choose all that apply.
X__ Y__ Z__
Appropriate response choices– Surveys can be frustrating when the questions are
fixed response without appropriate answer choices.– Provide answer choices such as “Don’t know,” “N/A,”
“Unsure,” and “Other” where appropriate.
X Does the laptop lending program meet your needs?Yes__ No__
Appropriate response choices– Surveys can be frustrating when the questions are
fixed response without appropriate answer choices.– Provide answer choices such as “Don’t know,” “N/A,”
“Unsure,” and “Other” where appropriate.
X Does the laptop lending program meet your needs?Yes__ No__
O Does the laptop lending program meet your needs?Yes__ No__ N/A__ [or “I’ve never used this program__”]
Questions or comments?
SURVEY TOOLS
PollWhat survey tools have you used? (check all that apply)– Google Forms– Paper-based– Poll Everywhere– Qualtrics– Survey Monkey– Other (let us know in the chat box!)
Survey toolsThere are many tools! I’ll mention a few, feel free to share your experiences in the chat• Google Forms• Qualtrics• Survey Monkey • Poll Everywhere• Paper!
Google forms
Google forms
Google forms
Qualtrics
Qualtrics
Survey Monkey
Survey Monkey
Poll Everywhere
Paper surveys
Image: http://www.educationnews.org/wp-content/uploads/2015/08/paper.jpg
Questions or comments?
ACTING ON SURVEY DATA
Involve your colleagues
Image: http://www.flickr.com/photos/dukeunivlibraries/
Share your results• Short email with key points about the results• Presentation for key staff or departments or
even the entire library• Brief report, sent only to the staff most
interested in the results• Links to the data for staff to explore on their
own
Code free text responses
Crowdsource the work!
Outreach opportunities
“Funding for students to purchase articles that Duke Libraries doesn't have access to would be nice. Often I find current articles that I can use for research but they are over $60.”
“It would be nice to be able to return Lilly Library DVDs to Perkins & Bostock.”
Triangulating data• Methods to consider when triangulating data:– thoughtfully planned focus groups– semi-structured interviews – observational studies– targeted, more focused surveys – usage statistics or other numerical data (e.g., gate
counts, circulation stats, web metrics)
Follow-up focus groups
Image: http://www.flickr.com/photos/dukeunivlibraries/
Reports, presentations, dashboards
Tableau dashboards: bit.ly/SurveyDashboards-2016
Staff workshop
Chart: emba.mit.edu/images/uploads/Impact_Matrix_MIT_EMBA.jpg Image: http://www.flickr.com/photos/dukeunivlibraries/
Staff workshop
Chart: emba.mit.edu/images/uploads/Impact_Matrix_MIT_EMBA.jpg Image: http://www.flickr.com/photos/dukeunivlibraries/
Organizing all of the great ideas
Organizing all of the great ideas
• Potential expenditures
Organizing all of the great ideas
• Potential expenditures• Service improvements
Organizing all of the great ideas
• Potential expenditures• Service improvements• Marketing opportunities
Organizing all of the great ideas
• Potential expenditures• Service improvements• Marketing opportunities• Assessment opportunities
…and following through
Sample projects
• Library e-newsletter • Dry erase marker kits • Low-cost earbuds• Posting hours to the
coffee shop• Improved signage • FitDesks
Process from start to finish 1. Conduct survey2. Analyze data 3. Share findings4. Develop recommendations5. Prioritize according to ease of
implementation and potential impact6. Assign ownership7. Follow through
Questions or comments?
KEY TAKEAWAYS
Involve your colleagues…
Image: www.flickr.com/photos/dukeunivlibraries/
and users…
Image: www.flickr.com/photos/dukeunivlibraries/
Form follows function
Plan to analyze
Test early and often
Triangulate methods
Image: https://commons.wikimedia.org/wiki/File:Japanese_Map_symbol_(Triangulation_point).svg
Act on what you learn
Image: http://www.flickr.com/photos/dukeunivlibraries/
Thank you!ALA eLearning will send participants a link to the recorded
workshop and slide deck.
With questions about our content, contact – Emily Daly: [email protected]– Joyce Chapman: [email protected]
With questions for ALA eLearning, contact– [email protected]