evaluating and assessing library instruction examples & best practices

Evaluating and Assessing Library

Instruction

Examples & Best Practices

Presenters

Martin Garnar – Regis University Rhonda Gonzales & Eleni Adrian –

CSU-Pueblo Lorrie Evans – Auraria Library Patrick McCarthy - CSU

Evaluating and Assessing Library Instruction

Regis University

Martin Garnar

ACRL Guidelines for Evaluation and Assessment

program evaluation plan criteria for evaluation learning outcomes and assessment methods coordination of assessment with teaching

faculty data gathering and analysis


CSU-Pueblo

Eleni Adrian & Rhonda Gonzales

CSU-Pueblo Library Instruction Evaluation

Overview

Librarians at CSU-Pueblo administered a brief evaluation form after each instruction session.

Collected these evaluations from 1999-2004. Entered results into an Access Database. Data Clean-up Statistical Analysis Conclusions


Evaluation Form

Background questions

Satisfaction questions

Ranking question


Evaluation Forms

Collected Fall 1999 through Summer 2004.

181 classes

2441 evaluation forms


Access Database

Entered several semesters’ of data

Revealed errors in our database design

New database structure

New input forms


Access Database


Data Fixes: New Database Design

Relationship between the classes and the evaluations tables.

No answers for background questions.


Data Fixes: Form Changes

Two questions added, two questions removed.Re-entered all old evaluation form data to

match the final version.Double-checked that the form changes were

accurately re-entered.


Data Fixes: Data Integrity

Duplicate Class records

Duplicate and blank Evaluations records

Typos in Status (year in school)

General data entry errors


Results – Background Questions

20.1%sophomore

16.0%senior

20.6%junior 3.0%

graduate

39.5%freshman

0.7%faculty

Category

seniorsophomore

facultyfreshmangraduatejunior

Status of RespondentsYear in School


Attended Before

Count

YesNoNA

1600

1400

1200

1000

800

600

400

200

0

973

1436

32

Attended Session Before


Attended Previous Session


Had a Specific Assignment

Count

YesNoNA

2000

1500

1000

500

0

1800

579

611

Student Had a Specific Assignment


Specific Assignment


Professor Present

Count

YesNoNA

2000

1500

1000

500

0

1948

349

144

Professor Participated in Session


Professor Participated


A

Count

654321*

1200

1000

800

600

400

200

0

1194

952

243

1714147

This presentation helped me learn how to start my research.

1=strongly disagree 6=strongly agree

Results – Satisfaction Questions

Start Research


B

Count

654321*

1400

1200

1000

800

600

400

200

0

1374

832

193

26763

The presentation was clear.



Clear Presentation


C

Count

654*312

1200

1000

800

600

400

200

0

1173

836

282

64501917

The hands-on experience will help me to complete my assignment.



Hands-on Experience


D

Count

65*413

1800

1600

1400

1200

1000

800

600

400

200

0

1680

361338

5174

The librarian treated each student with respect.



Respect


15.1%5

15.8%4

8.9%3

24.1%2

36.1%1

Category

5

1234

The most important thing that I learned was:

1=how to locate a journal article or book2=how to access the library's databases3=how to use Interlibrary Loan to get my information4=how to decide which databases I should use5=how to search the Internet more efficiently

Most Important Thing Learned

Results – Ranking Question


Comparisons

Significance of Having an Assignment on Students' Feeling that Presentation Helped Them Start Their Research

0.00%

50.00%

100.00%

No Assignment Had Assignment All Respondents

No Assignment 0.52% 1.22% 1.39% 11.98% 43.75% 41.15%

Had Assignment 0.61% 0.39% 0.50% 9.41% 37.79% 51.31%

All Respondents 0.59% 0.59% 0.72% 10.03% 39.23% 48.84%

1 2 3 4 5 6


Significance of Having an Assignment on Students' Feeling that Hand's-On Practice Was Helpful

0.00%

20.00%

40.00%

60.00%

No Assignment Had Assignment All Respondents

No Assignment 1.42% 1.06% 2.48% 15.93% 35.22% 43.89%

Had Assignment 0.57% 0.63% 2.00% 10.61% 34.91% 51.28%

All Respondents 0.78% 0.73% 2.11% 11.91% 34.99% 49.48%

Strongly Moderately Slightly Slightly Agree Moderately Strongly Agree

Comparisons


Conclusions

Distribution of responses skewed towards high end.

Student evaluations of faculty teaching across campus are not normally skewed towards the positive to this extent.

These results show that a large majority of students felt that our instruction sessions were helpful!


Conclusions - Meeting Our Goals?

Since most of our sessions are introductory in nature, one of our primary goals is to show students how to start their research using our library.

88.17% of students that took the survey indicated that they moderately or strongly agreed that the sessions helped them learn how to start their research.

36.12% of students who took the survey responded that learning how to locate a journal article or book was the most important thing they learned.


Conclusions - Factors Affecting Results What factors may have affected student responses?

Chi-square tests indicate that the following factors proved to be statistically significant:

Having a specific assignment in conjunction with this presentation was a significant factor on the number of students that strongly agreed with A & C

Attending a prior session was a slightly significant factor in determining how many students strongly disagreed with D

Survey Design: Even number of response choices forced students to choose positive or

negative Unclear wording – “My Professor participated in this session.” Lack of N/A forced students to choose a response even if it did not apply

Survey Administration Administered at the end of each session, when students were in a hurry to

leave. Tendency of librarians may be to skip the survey if the session hasn’t gone

well.


Indications for Best Practices

Role of Evaluation:combine learning assessment with program evaluation

Survey Design:concise, clearly worded, avoid mild statements, allow neutral and N/A responses, define hypotheses prior to creating the form


Auraria Library

Lorrie Evans

Evaluation of instruction at Auraria Library

Survey student learning in single session library classes (random selection).

Survey students in full semester credit bearing classes (assignments, final project)

Survey the faculty toward the end of the semester – after papers and assignment are turned in.

Do the students understand the most important concepts?

No other chance for a follow up or review.

When students leave the room we will not see them again

Challenges in the single session library class

Look in the mirror – are we teaching what we think we are teaching.

Collaborate with campus faculty on a pro-active level.

Make library instruction a more integrated part of the academic culture.

Accreditation – assessment mandates driven externally.

Why?

The tool – short online survey

In class – three to five questions mapped to learning goals.

Customized for each class

Broadcast results, review answers & main concepts.

Students share concerns.

•Students see their answer along with classmate responses. •Librarian can respond to student input.•Review main points.

Sample results

English 1020 Student comments view

Psychology freshman seminar data

Graduate Political Science - single session post test.

Graduate English Literature - semester course

Jump past results

Comments

return

Freshman Seminar

sample items

Psych

return

Graduate English Literature survey:sample items

return

post test

Political Science

return

Right tool for the job question

Faculty survey -- end of the semester

Program effectiveness as reflected in quality of student projects and assignments.

Satisfaction with content, scheduling and teaching.

Faculty Evaluation

survey results

Zoomeranglink

Faculty evaluations

sample items

How can we improve?

Feedback loop

In class survey – instant feedback for both students and librarian. Librarian can go over confusing concepts.

Results discussed at departmental meetings – used as source of information to make adjustments in teaching (more depth with less quantity).

Results made available to instructors and students.

End of the year report submitted to the campus Outcomes Assessment Advisory Committee. The committee provides a review of the report, recommends improvements.


CSU – Fort Collins

Patrick McCarthy

Evaluating Learning Outcomes at the Reference Desk - CSU

Why? Determine if learning takes place as a result of

interactions at the reference desk Identify those skills most suitable to reference

desk instruction Establish reasonable reference desk learning

outcome benchmarks Recommend behaviors and methods that lead to

student learning

Categories of Reference Assessment

Accuracy response rate User satisfaction Course/instructor opinion survey Learning outcomes

Methodology?

Quantitative approach Powerful, not practical

Qualitative approach Powerful, versatile

Procedures

1. Patron completes preliminary survey at completion of reference interaction

2. Researcher analyzes preliminary surveys and selects relevant patrons for follow-up assessment

3. Selected patrons are interviewed on specifics of reference transaction

4. Interview recordings are transcribed and coded for assessment

Analysis

Transfer assessment Accuracy assessment Patron must provide evidence Authentic assessment

Group Discussion – Guiding Questions

1. How have you determined the learning needs of your community?

2. What's your best method for evaluating library instruction?

3. What's your best method for assessing progress on learning outcomes?

4. Do you have examples of successful collaboration with classroom faculty for evaluation and/or assessment?

5. How have you used data from your instruction program evaluations and/or assessments?

Assessment of Instruction – Examples & Best PracticesPresenter Contact Information

Martin Garnar, Reference LibrarianDayton Memorial Library, Regis University3333 Regis Blvd., Mail Stop D-20Denver, CO 80221-1099303.964.5459303.964.5497 (fax)

[email protected]

Lorrie Evans,Library Instruction ServicesAuraria Library1100 Lawrence StreetDenver, CO 80204303.556.6776

[email protected]

Patrick McCarthyCoordinator of InstructionMorgan LibraryColorado State Univ.501 University AvenueFort Collins, CO 80523720.491.1881

[email protected]

Rhonda Gonzales, Interim DeanEleni Adrian, Instruction CoordinatorUniversity LibraryColorado State University – Pueblo2200 Bonforte Blvd.Pueblo, CO 81001719.549.2361rhonda.gonzales@[email protected]

evaluating and assessing library instruction examples & best practices

Documents