continuous assessment = continuous improvement vale users conference, 1/9/09 mark thompson,...
TRANSCRIPT
Continuous Assessment = Continuous Improvement
VALE USERS CONFERENCE, 1/9/09
Mark Thompson, Assistant Director for Patron Information Services
Sidney Silverman Library Bergen Community College
Paramus, NJ
Sidney Silverman Library 2
Simple – Yes??
Sidney Silverman Library 3
It’s Needed More Than Ever
Assessment is needed Academic environments are
changing; User dynamics are changing; Resource and content platforms are
changing; Accountability demands are
increasing; Funding sources are under stress.
Sidney Silverman Library 4
Working Premise
Basic Premise If you continually assess,
then you will continually find ways to improve.
If you assess the core issues, then the core problems will be
relieved.Versus If you assess sporadically,
Then you will improve sporadically. If you assess urgent problems,
then key priorities will be neglected.“Tyranny of the Urgent”
Sidney Silverman Library 5
Working Assumptions
We Need to address BOTH active assumptions –
Assumption 1: To be effective, assessment is iterative; Research Identify issues - Address issues
Research Identify remaining issues address again Research …………
Assumption 2: And, perspectives must be active. Do you have active statements for?
Mission / Values / Goals / Strategies If so, they can lead you to identify important
priorities, assessments, and actions.
Sidney Silverman Library 6
But, Did Any Of It Work? YES! More usable web pages
Driven by usability studies Effective “Quiet Library Campaign”
Developed a logo and displays Better first step: tour and tutorials Benchmarks for library instruction
Tracking of unique users Informed decisions on renovations
New customer service initiatives
Bergen Community College
LIBRARY WEB: Usability Studies
Thought of as many ways as possible to get user input (as quickly as possible) Students first
Including Student Ambassadors Librarians as superusers AND experts Teaching faculty and administrators
Adhoc: 1:1, 1:2, and small groups Tied to class schedules At Library terminals
What did NOT work: Testing at end of library instruction classes
Our New Logo!
Developed with input from students and staff with a focus on the positive traits necessary for success in college.
Sidney Silverman Library 9
Display: Reclaim the Quiet
Sidney Silverman Library 10
OK, but how do I do this?...
Take the strategic planning plunge; Adopt an assessment framework --
any framework! Note your progress and keep
asking users: over the long term. Ask in different ways, at different
times. Track your on-going issues and don’t
give up.
Sidney Silverman Library 11
Getting to THE Idea
Sidney Silverman Library 12
Vantage Point: Strategic Planning
Strategic planning framework
Priorities of what toaddress
Available research techniques
ASSESSMENT
Strategic Planning Model
© Bruce Flye—Society for College and University Planning (SCUP)
Strategic Planning Process - Details
OUTPUTS
YEAR 1 YEAR 2 YEAR 3STAGE 1: Planning and Mission
Planning *Obtain support for re-assessment.*Develop wide base of involvement.*Publicize planning efforts.
Communications to college communitiesStrategic Plan 2005: Implementing a Planning Process
Values re-assessment *Discuss organizational values.*Identify values that will drive decisions.*Collegewide open forums.
included as part of Vision Statement
Review vision and mission
*Review the why, what, how and for whom (vision/mission).
College Vision and Mission Statements.Dept Vision and Mission Statements.
STAGE 2: SWOT
SWOT Analysis *Analyze: Internal (Strengths, Weaknesses)and External (Opportunities, Threats)*Gather input, incl Listening Sessions; Public Forums*Conduct gap analysis
SWOT ReportDocument sources used for external data
STAGE 3: Strategic Issues
Strategic Issues and Priorities
Using SWOT/gap analysis form strategic issues.Make strategic decisions and form priorities.
Readout from public forumsCollege Strategic Priorities to 2013
Challenges & Constraints
Examine issues and blocks that prevent movement toward vision
ON-GOING ACTIVITIES
Environmental scanning Environmental scanning Environmental scanning Institutional Research reports
Operationalize Strategic Priorities (Action Plans)
Operationalize Strategic Priorities (Action Plans) Operationalize Strategic Priorities (Action Plans)
College 3-year Action PlanAnnual report -goalsBudget reports
Annual reporting and review Annual reporting and review Annual reporting and review Progress Report on Strategic Priorities
Conduct assessment Conduct assessment Conduct assessment 2-year Cycle Assessment Report
CIE Advisory Board Overview CIE Advisory Board Overview CIE Advisory Board Overview
Communicate Efforts to College Community Communicate Efforts to College Community Communicate Efforts to College Community
Bergen Community College - Sidney Silverman Library
15
Background
Bergen Community College, Paramus, NJ 16,000 students; 50% FT/50% PT 140 different countries represented Library: open 88 hours/week;
4,000 gate count per day
Library Instruction Program 275 sessions(80 min. ea.) per semester
9 librarians teach; 2 classrooms (24 PCs ea.) Learning objectives written for: ALP (i.e. ESL), EBS
(basic English), Comp I, and Comp II
Sidney Silverman Library 16
Bergen Community College
We Adopted Assessment Framework College-wide effort on strategic planning
and for assessment: Built by team which included a library manager.
Began fall ’05 3 ½ years of effort
And, We Ask Our Users Often LibQUAL surveys: 2005, 2008 Customer suggestions / feedback forms
Other adhoc emails and surveys Qualitative research
1:1, adhoc groups, expert groups Student Advisory Group
Sidney Silverman Library 17
Example Framework
A Framework for Assessment 6 Key Elements:
1. Choose your outcome (goal);2. Relate to College mission/goals;3. Select means of assessment;
1. Methodology and data sources
4. Identify criteria for success;5. Summarize data collected;6. Describe how results will be used.IMPLEMENT
Sidney Silverman Library 18
Relevant Section of College Mission & Goals:(In this area list the relevant section of the College’s Mission and Goals)
Your Department or Unit Mission:(State your unit’s mission statement.)
BERGEN COMMUNITY COLLEGE Department or Unit: Year:
Intended Departmental Outcome #1: (must be explicitly related to
College’s M & G)
Means of Assessment and Sources of Data:
Criteria for Success:
Summary of Data Collected:
Use of Results:
Intended Departmental Outcome #2: (must be explicitly related to the
College’s Strategic Plan)
Intended Departmental Outcome #3 :(goal related to Core
Competencies, if applicable)
Intended Departmental Outcome #4:
A Framework for Assessment (BCC, Fall 2005)
(1) (2) (3) (4) (5) (6)
Sidney Silverman Library 19
It does help.
Sidney Silverman Library 20
Assessment’s Continuous Loop
Strategic Priorities Select
Goal to Assess
Set Criteria for Success
Which Measurem
ents?
Conduct Research
Analyze Results
Develop & Apply
Remedies
Sidney Silverman Library 21
Specific Example in Using Framework
Example: Second Assessment Cycle Columns 1 & 2 2005-2006 Assessment Goal: Library First two elements (from Framework):
Unit Outcome #1 (Column 1) Reduce noise in the Library.
Related College Statement (Column 2) “To provide supportive services and leadership
opportunities in an environment that enables and encourages students to achieve…” from Goal #2, Missions and Goals
Sidney Silverman Library 22
Example, con’t.
Columns 3 & 4 Means of Assessment (Column 3)
Analysis of LibQUAL data, Spring ‘05 Focus groups/interviews: users and staff,
Summ’05 Instituted a Comments Box for feedback, Fall ’05 Conduct a follow-up survey, Winter ’05-’06
Criteria for Success (Column 4) Identify key problems with noise. Achieve a consensus over how to impact noise
issues. Develop policies and methods to improve.
Launch a “Noise Campaign” by 9/1/05. After which: a majority of users find the
library acceptably quiet.
Sidney Silverman Library 23
Example, con’t.
Column 5 Summary of Data (Column 5)
Observations and complaints (BLOG, email, in-person), show noise problems, Fall ‘04
LibQUAL survey, April 2005 “Library as Place” rated the lowest
Staff and user meetings, Summer 2005 Identified possible solutions
Follow-up assessments after Noise Campaign
Show higher satisfaction w/quiet
Excerpts: Summary of Data
LP-2 Issue of Noise (Perceived and Desired Means)Scale of 1 low and 9 high
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
Perceived
Desired
Excerpts
LibQUAL: “Library as Place” ratings were low Comments:
Noise levels too high; Lack of enforcement; Need more group and classroom areas
Feedback Forms Gave Us the Specifics-- Library is Too Noisy
Students talk loudly and are inconsiderate. Groups form and talk. Everyone uses a cell phone. Librarians do not enforce the rules. Noise comes from classrooms.
Example, con’t.
Use of Results (Column 6)Quiet Campaign launched 9/1/05
All areas designated either Silent or Quiet zones Issues identified and Warning Cards handout out to students: Awareness builders: “yakker trackers,” flyers, & bookmarks
Follow-up survey done Oct – Dec. 2005 475 Comments Slips received; 46% gave us specific comments Good Results:
65% said it was quieter than last semester63% said it was now quiet enough for them
Sidney Silverman Library 27
Other Assessment Efforts at BCC
Capture Individual user comments about any aspect of Library services or place
Compare year to year Study Library Instruction
Satisfaction survey: faculty and students Unique user tracking Student perspectives on topics and
teaching Establish benchmarks of customer
service How are we doing?
Sidney Silverman Library 28
Use a Variety of Measures
What we are using already
REF desk log stats LibQUAL
Feedback forms Email surveys
Suggestion box Student Advisory Group
Drop in suggestions Web site suggestion links
What we are planning
BI effectiveness (pre/post)
Query of new satellite campus users
BI tracking survey Web site usability team
LibQUAL 2011 Customer service survey
So far: Individual User Comments
Two survey efforts yielded a total of 1,290 user comments & suggestions
From 2005 LibQUAL = 200 comments (43% of 468 total
respondents) 549 Feedback Forms
From 2008 LibQUAL = 287 comments (38% of 758 total
respondents) 254 Feedback Forms
Helped Us ID: Concerns & Issues
Aggregate comments indicate some problems persist.
These are: availability of computers, noise, and customer service.
These concerns run as a constant. Since 2005 in over 1,290 comments from
LibQUAL 2005, Feedback Forms, and LibQUAL 2008.
Bergen Community College - Sidney Silverman Library
31
What to Cover in Library Instruction
How to Pick a Topic
How to Get Library Help
Evaluating Internet Sites
Library Homepage Links
How to Find Books
Finding Articles in the Library Databases
Advanced Internet Searching
Accessing Library Resources from Home
122 123 129 104 124 141 135 128
69% 70% 73% 59% 70% 80% 77% 73%N=176
What Students Want Covered in the Session (simple check-off, Yes/No)
lowest highest
32
Libr Instr: More Lecture or Hands-on?In an ideal 80-minute library instruction class, what percent of time should be spent in lecture/demonstration versus hands-on practice?
0
10
20
30
40
50
60
700-
9
10-1
9
20-2
9
30-3
9
40-4
9
50-5
9
60-6
9
70-7
9
80-8
9
90-9
9
Perferred % of Lecture/Demo (vs. Hands-On)
Num
ber o
f Res
pons
es
Students Faculty
Benchmarks & Comparisons Comparisons:
Between pre-renovation survey in 2005 and post-renovation in 2008
Library usage is higher. 68% of users come in daily or weekly 53% of users go to Library website daily/weekly.
Overall satisfaction remains high. On 1 to 9 (high) scale, users are satisfied with:
“Overall quality of the service provided in the Library.” 7.43
“The way I am treated at the Library” avg. 7.5
Benchmarks & Comparisons
Good overall results, but some causes for concern:
Statement: “Giving users individual attention” is lowest rated of all factors.
Disaffected or non-users: 14.3% rarely or never use library 24.1% rarely or never use library’s website
And on Feedback forms: “Did you accomplish what you set out to do?” and
“Was the level of service received sufficient?” -- # of positive responses is decreasing.
Sidney Silverman Library 35
So…. What’s Next Take action by:
Adopting a strategic planning model Take on an assessment framework Use a variety of measurement tools Follow a cycle
Document the results Conduct constant follow-up
Assess implemented improvements Benchmark and compare
Implement change and Begin again…
Sidney Silverman Library 36
Contact
Thank you! Questions? Mark Thompson
Asst Dir, Bergen Community College Paramus, NJ 201-447-7447 [email protected]
Sidney Silverman Library 37