G A I N S P R I N G M E E T I N G , M E R C E R U N I V E R S I T Y , M A C O N , G AA P R I L 2 4 , 2 0 1 4
MEASURING YOUR IMPACT: USING EVALUATION FOR LIBRARY ADVOCACY
3
AGENDA
Morning
9:00 Introductions and Overview of Workshop
9:20 Demonstrating the Library’s Value – Mission & Vision
9:50 The Environment (SWOT exercise)
10:10 The Plan - Logic Model
11:00 Break/Group exercise
11:30 The Plan – EvaluationCreate your own, share
Afternoon
12:00
LUNCH
1:00 Data, Tools, and Analysis
2:00 Communicating the Value
3:00Create your message – elevator speech, infographic, annual report introduction
3:45 Resources, class questions and evaluation
4
WORKSHOP OBJECTIVES
By the end of the workshop you will:• Understand your library’s value in terms of the
mission of the larger organization• Describe some tools used to assess your library,
its users and stakeholders• Identify the tools and methods used for data
collection and analysis• Design an evaluation plan for a service in your
library• Communicate evaluation results
5
WHAT THIS WORKSHOP IS NOT…
• Custom-tailored• A quick fix• One size fits all
6
“WHAT GETS MEASURED, GETS DONE.“
Proving to your administration that money spent in your library is going to pay student achievement dividends is well worth the time and effort
http://www.doug-johnson.com/dougwri/what-gets-measured-gets-done.html
7
DEMONSTRATING YOUR LIBRARY’S VALUE
8
WHY DEMONSTRATE VALUE?
• To show the impact of your library on the larger organization’s mission and goals
• To show accountability for your resources; your library’s contribution to the bottom line
• To demonstrate that your library provides value-added services
• To use as an advocacy and marketing tool
9
HOW & WHAT TO EVALUATE
• What you decide to evaluate depends on:• What you need or want to know• What your users feel is important• What certain stakeholders want to have evaluated
• Use “smallball” evaluation• Base hits or home runs?
10
LIBRARY SERVICE IS VALUE-ADDED
• Cost savings• Institutional & Personal
• Time savings• Travel, discovery, skills acquisition, opportunity for other
uses of time
• Access• Opportunity to use/apply information
• Impact on learning, research
11
THE BOTTOM LINE
• You are either generating revenue, or supporting those who do
OR …
• You are helping to control operating expenses or supporting those who do
OR …
• You are creating expenses that add recognized value
OR …
• You are creating expenses that must be controlled or eliminated to reduce overhead
Will Welton, PHD, Director, MHA Program, University of Washington
12
THE BIG QUESTION!
What is the bottom line and what is your library doing to support it?
13
WHAT YOU NEED IS A PLAN
• Know the vision• Look at the environment• Create the plan• Collect and analyze the data• Communicate the value
14
THE VISION
15
KNOW WHAT YOU WANT TO DO AND WHY
• Align your library’s mission & goals with the organization’s mission & goals
• Your library’s value = library’s contribution to achieving organization’s mission & goals
16
MISSION STATEMENTS
• dedicated to cultivating the inherent gifts, skills and talents of children in Grades 6 through 12 so that they are prepared for success in college and life.
• to provide resources that support the curriculum, help our students to develop habits to become lifelong learners and self-actualized citizens, and promote understanding and appreciation of our Marianist heritage.
17
MISSION STATEMENTS
• ...foster in our students academic, physical and creative fulfillment, together with strength of character, while helping them become productive members of our school community.
• …support and … foster the active and creative pursuit of academic and artistic achievement through the effective use of information and the encouragement of curiosity and scholarly inquiry.
18
HAIKU
creative fulfillment,academic and physical;productive community
...foster in our students academic, physical and creative fulfillment, together with strength of character, while helping them become productive members of our school community
effective information useencourages curiosity andscholarly achievement
…support and…foster the active and creative pursuit of academic and artistic achievement through the effective use of information and the encouragement of curiosity and scholarly inquiry
19
WRITE A HAIKU
• Write your mission statement or another message• Turn that into a haiku• Standard haiku is 5/7/5 syllables, but English
haikus are more flexible because of our syllable structure
• Share your haiku
20
MADE TO STICK
• SUCCESs• Simple – core, compact• Unexpected – gets attention• Concrete – specific, not abstract • Credible – testable• Emotional – more than self interest• Stories – stimulate, inspire
(From: Made to Stick. Heath, Chip. New York: Random House, 2007)
THE ENVIRONMENT
21
22
WHY LOOK AT THE ENVIRONMENT?
• Understand needs, desires and problems in context
• Validate assumptions about your contributions and services
• Provide a baseline for future evaluation• Help to develop the blueprint to plan and
evaluate your contributions and services
23
COMPONENTS OF ENVIRONMENT
• The organization• Your library• Clients/users• Stakeholders• The community
24
ASSESSING THE ENVIRONMENT
• User/stakeholder input• Surveys, focus groups, interviews• Unsolicited feedback• Conversations
• Observation• Assessment by walking around• Know the language of your users
• Information/literature• Journal/serial/magazine
25
ASSESSING THE ENVIRONMENT
• Library statistics and records• Collection use• Librarians’ work including reference, teaching, etc.
• Get out of your library and talk to people• Understand the role of information in their work• Learn how information is acquired, applied• Discover information problems they are trying to solve
• SWOT analysis• Understanding strengths, weaknesses, opportunities,
threats• Consider how these affect your work, your assessment
26
LIBRARIANS’ TOP 10 TASKS
• School Librarians say…1. Help students select books2. Collaborate with teachers3. Read and book talk with classes4. Teach research and use of technology to students and teachers5. Keep library organized including cataloging, weeding,
inventory, late notices, vacuuming, dusting6. Study standards to prepare library lessons7. Troubleshoot technology issues around the building8. Help with extracurricular activities including laminating, school
plays, lunch duty, etc9. Prepare book, equipment and supply orders10. Organize special library events like book fairs, author visits,
contests
27
LIBRARIANS’ TOP 10 TASKS
• School Principals say…1. Help students access information and books2. Help faculty access information and books3. Share technology expertise with students and teachers4. Select “appropriate” materials5. Model love for reading6. Collaborate with teachers7. Provide equipment and technology8. Provide leadership with technology9. Teach research skills, teach about books and databases10. Provide an inviting environment
How does your boss see you? Proof that principals value librarians. School Library Journalhttp://www.slj.com/2012/09/careers/how-does-your-boss-see-you-proof-that-principals-value-librarians/#_
28
• Librarians say
• Principles say
29
SWOT ANALYSIS
Weaknesses Internal
◦ Statements about what is lacking in your library
Opportunities External
◦ What do stakeholders want you to do that you are not doing?
Threats External
◦ Factors that can adversely impact your library’s goals
Strengths Internal
◦ Positive statements about your library
30
PLAN BACKWARD, IMPLEMENT FORWARD
31
THE LOGIC MODEL - A TOOL TO…
• Help organize your thoughts – both before and during a project/program
• Provide a framework for planning and evaluating programs
• Clarify intended outcomes
• Acts as a communication tool
Good source for more information:W.K. Kellogg Foundation Logic Model Development
Guide (pdf on your thumb drive; see class bibliography for
link)
32
PIECES OF THE LOGIC MODEL
• Outcomes• Activities• Resource• Outputs
33
OUTCOMES
• Outcomes - changes in attitude, behavior, skills, knowledge or situation• Short, medium and/or long term• Intended or unintended• Positive as well as negative• S.M.A.R.T.
34
SMART OUTCOMES
Outcome: Health resources Web pages are available on the library website by the end of the project
• Specific: one or more web pages with links to EBM resources are available on the library website
• Measurable: done or not; do nurses access resources
• Action-oriented: created and posted; nurses access resources
• Realistic: It is possible within the timeframe
• Timed: “when” is articulated
35
OUTCOMES
Outcome• Long Term:
• Patients and their families will have improved health information literacy
• Intermediate Term:• Staff will have information for timely clinical decision-making
• Short Term:• Health resources Web pages are available on the library
website by the end of the project
36
ACTIVITIES AND RESOURCES
• Activities • What will you do?
• Resources• What you have
• Income• Equipment• Collection
• What you need• Operating expenses (e.g., personnel, acquisitions,
maintenance, etc.)• Funds for new initiatives or services
37
OUTPUTS
• Outputs• How many did you do?• How many attended?• How many were distributed?• How many times was it used?
38
Resources
Activities/Outputs
Outcomes
If we get theseresources…
Conduct theseactivities and deliver these products
We will see these outcomes
PUTTING IT ALL TOGETHER
39
THE MODEL MAY CHANGE OVER TIME
Activities
OutputsOutcomes
Resources
40
LOGIC MODEL
Resources Activity Outputs Outcomes
41
LOGIC MODEL
Resources Activity Outputs ST/IT/LTPersonnel, money, expertise needed
What you will do What your activity will produce – data, classes, brochures, etc.
The SO WHAT & WHEN– the benefits that accrue as a result of your program
Knowledgeable librarian Select web resources List of URLs and descriptions of resources/sites
HTLM expertise Create web page(s) Write HTML code or edit a template/existing page(s)
Health resources Web pages are available on the library website by the end of the project
Tracking software Collect page hit statistics; survey staff about usefulness, timeliness of web based information
Web statistics; survey results
Staff will have information for timely clinical decision-making
Outcomes: ST: Health resources Web pages are available on the library website by the end of the project; IT: Staff will have information for timely clinical decision-making
42
LOGIC MODEL GROUP ACTIVITY
• Identify outcome(s)• Articulate short/intermediate/long term• List
• activities • outputs• resources
43
THE EVALUATION PLAN
44
THE EVALUATION PLAN
• Builds on the logic model• Evaluates success• Assesses value
45
CONSIDER THE PURPOSE
• Who is your audience? This may be different than the beneficiaries of your services• Your users• Administrators• Patients/families
• How will the information be used?• Financial savings or justification• Intangible or non-monetary value of program
benefits to community• Marketing and advocating for your library
46
WHAT DO YOU WANT TO KNOW?
Have you achieved your stated outcome?• Short Term:
• Health resources Web pages are available on the library website by the end of the project
• Intermediate Term:• Staff will have information for timely clinical decision-making
• Long Term:• Patients and their families will have improved health
information literacy
47
WHEN DO YOU WANT TO KNOW?
• Consider objective(s)• Articulate the time frame
48
EVALUATION PLAN
Activity Indicator
Target Data Source
Data Analysis Methods
Data Collection Frequency
What is done to achieve outcome – from Logic Model
Measurable result of activity
Specific indicator measurement desired
Origin of indicator measurements
Organize, examine, learn from the data
Date, time and intervals
Outcome (from logic model):Staff will have information for timely clinical decision-making
49
EXAMPLE PART 1
Activity Indicator Target
What is done to achieve the outcome
Definition of success
Specific desired result
Create web pages; analyze use statistics; survey potential users
Web pages are available; Users indicate the information was available for clinical decision making
50% of respondents report information was available for timely [define timely here] clinical decision- making
Outcome (from logic model):Staff will have information for timely clinical decision-making
50
EXAMPLE PART 2
Data Source Data Collection Frequency
Data Analysis Methods
Origin of indicator measurements
Date, time and intervals
Organize, examine, learn from the data
Web statistics, survey
Data recorded quarterly throughout fiscal year; twice annual survey
Spreadsheets; survey coding
51
NOW DESIGN YOUR OWN EVALUATION PLAN
52
FROM YOUR LOGIC MODEL…
• Select an outcome and describe:• An activity that will help achieve the outcome• Indicators of outcome results• Targets for indicators• Data source• Data collection frequency• Data analysis methods
53
MAKING SENSE OF THE DATA
54
ANALYSIS
• Think about data analysis before collecting the data
• Identify experts that can help with data analysis• Test your collection and analysis tools• Leave enough time• Go “smallball” if needed
55
YOUR DATA
• Quantifiable• Statistics
• What do you collect?• What do you do with it?
• Evidence of change• Measurable impact of services provided
• Qualitative• Stories, opinions• Evidence of change• Perceived impact of services provided
56
SOME TOOLS
• Benchmarking• How you compare to others of similar size and
circumstance• A common measuring stick to evaluate process
performance • Qualitative/quantitative analysis
• Coding narratives• Scoring responses
• Cost/Benefit Analysis• Use to evaluate the benefits of a program or service
• Return on Investment• How much your investment in the program or
project earned for the company
57
BENCHMARKING
• Can improve your library's performance• Can help you gain upper management support• Can help prove the value of your library
58
BENCHMARKING RESOURCES
• Your own library
• The competition
• Other libraries in your system, industry
• Associations:• ARL – LibQual+• AAHSL – Annual survey• NN/LM MAR library value study
59
QUALITATIVE AND QUANTITATIVE
• Narrative• Code text for concepts• Determine frequency of concepts• Describe/illustrate results
• Quantitative• Assign values to possible responses• Calculate mean, median, mode of response values
60
CBA: COST/BENEFIT ANALYSIS
• The ratio showing dollar value of benefits gained for dollar value of costs
• Use to evaluate the benefits of a program or service
• Benefits divided by costs gives the value realized by transaction
61
CBA FORMULA
BENEFITS ÷ COSTS
Or
BENEFITSCOSTS
62
ROI: RETURN ON INVESTMENT
• Percentage showing the return (increase in value) on dollars spent to achieve a benefit• Amount the company’s investment in a program earned
for the company• Earnings on money spent
63
ROI FORMULA
((BENEFITS – COSTS) ÷ COSTS) × 100
Or
BENEFITS – COSTS COSTS
× 100
64
PREPARING FOR CBA AND/OR ROI
• Select the program or service to value • Journal subscriptions; searching the literature class
• Identify benefits derived from program/service • Users have access to literature for decision making
• Identify who gets the benefit • Library users, patients, care team
• Convert the benefits to quantifiable terms (based on surveys and/or records)
• # of journals used; # of articles accessed; time available for research, patient care
65
EXAMPLES OF BENEFITS
• Books used (in house or borrowed) • Print journals used (in house or borrowed) • E-journal articles accessed• Patron time saved• Value of information available for research,
teaching, patient care
66
COSTS
• Purchase price• Staff time• Space, light, heat, computers, benefits
67
COSTING TIPS
• Consider tangible vs. intangible benefits
• Be conservative in estimating benefits and liberal in assessing costs
• Obtain administrative approval of the value of the benefits for each alternative; involve financial staff
• Express benefits and costs in a value system shared by all parties – $$
• Think about inflation and depreciation if study extends over several years – only if the CFO insists!
68
A CBA/ROI EXAMPLE
Subscription to online Fictional Medical Journal
at Anytown Institution
69
JOURNAL SUBSCRIPTION
• The library spends $1,600 for an institutional subscription to Fictional Medical Journal (FMJ) online and $200 to manage the subscription
• 7,800 articles are viewed per year using the library’s subscription
70
ASSUMPTIONS
• Each person reads on average 2 articles per week• 2x52=104 articles per user
• 7,800 uses represents 75 users• 7,800/104=75
• Without a library, each person who needed FMJ would subscribe at a cost of $99 for online only
71
COSTS AND BENEFITS
• Cost: $1,800• $1,600 for subscription and $200 for staff time to
process/maintain
• Benefit: $7,425• 75 personal subscriptions: 75 x $99
72
CBA MATH
BENEFITSCOSTS
$7,425 = $4.125
$1,800
$4.13 benefit to the institution for each dollar spent by the library on FMJ
73
ROI MATH
$7,425 – $1,800
$1,800
= 312.5%
Net Costs
Benefits – Net Costs× 100
× 100
74
REPLACING YOUR LIBRARY OR A LIBRARY SERVICE
• What is the value of the resources and services your library provides?
• What would your users pay on the open market?• Would your institution pay the retail costs?• Would your institution pay for everyone who
needs information or for select staff?
Retail value calculatorhttp://nnlm.gov/mcr/evaluation/calculator.html
75
CALCULATORS THAT DO THE MATH FOR YOU
• Retail value calculator• http://nnlm.gov/mcr/evaluation/calculator.html
• CBA/ROI calculator• CBA: Benefits/Costs• ROI: ((Benefits-Costs)/Costs) *100
• http://nnlm.gov/mcr/evaluation/roi.html
76
77
CALCULATOR DATA POINTS
• Where do the values come from?• What does each line mean?• http://nnlm.gov/mcr/evaluation/valuation.html
78
Becker Library
79
80
81
CBA/ROI EXERCISE
• Select one service or resource• List all the costs associated with providing the
service once or obtaining one resource• List alternative sources for that service or
resource• List the costs associated with that alternative
source• Plug the numbers into the formula
82
COMMUNICATING THE RESULTS
83
USE THE RESULTS…
• To communicate your value• To promote and advocate for your library• To maintain or get additional funding• As baseline information to show improvement
over time• To improve library services and programs
84
SOME COMMUNICATION METHODS
• Written and/or oral reports• Formal • Informal
• Graphics
• Executive summary
• Press release for newsletter or local paper
• Promotional materials for your library
• Staff meetings
• Informal hallway or elevator talk
85
MADE TO STICK (REDUX)
• SUCCESs• Simple – core, compact• Unexpected – gets attention• Concrete – specific, not abstract • Credible – testable• Emotional – more than self interest• Stories – stimulate, inspire
(From: Made to Stick. Heath, Chip. New York: Random House, 2007)
86
87
88
89
91
TIPS FOR WRITING
• Frame message in terms of added value to the mission and goals of the organization
• Summarize in one paragraph then support the message with evidence.
• Use figures, tables, graphs to make your point• Should be able to stand alone• Have a narrative to give more details
• Have others review the communications
92
TIPS FOR TALKING
• Know and rehearse your message
• Talk to administrators one-on-one when possible
• Focus on benefits
• Don’t use library jargon
• Educate, but don’t bore
93
“ELEVATOR TALK”
• Simple, memorable mini-speech
• Ready for any brief encounter
• What your library contributes and why that matters
• Create curiosity
Gardner, Reed S. Making the Case for Your Library: A How-To-Do-It Manual. How-To-Do-It Manuals for Librarians, Number 104. New York: Neal-Schuman, 2001: 6
“Sadly and too often, the people who have the most influence over our future are really not all that interested in libraries.”
94
ELEVATOR TALK EXERCISE
• Select an outcome • Invent results• Determine the audience• Create
• a sound bite that highlights the value of the outcome to that audience
OR• an interesting but mysterious statement that will make
your listener want to know more
95
GO FORTH & EVALUATE
• Do smallball evaluation• Relate your value to the mission and goals of your
organization • Plan your evaluation before you begin• Share your value with many audiences• Don’t wait for a crisis – start now!