Instructor-Librarian Case Study Evaluation of an Online Embedded Librarian

Implementation

Mary Edwards, MLIS, AHIP, EdD; Erik Black, PhD Health Science Center Libraries and UF College of Medicine, Department of

PediatricsThe University of Florida

Agenda

• Background• Introduction• Context• Methods• Data• Implications• Questions & Discussion

Embedded Librarianship Literature

• Public Services Quarterly special issue 2010• Shumaker & Talley, 2009 – Special Library

Association• References in the paper• Good literature on implementation, not as

much on assessment

Purpose

The purpose of the research project was to:• Develop an embedded librarian program to

support students in the OnMed program.• Evaluate the program using a case study

approach.• Provide empirical evidence to support the

efficacy of embedded librarians programs.• Guide future embedded librarian

implementations and evaluations.

Overview of the Case

• Institution• Program• Course• Participants

Institution

UF Fun FactsEnrollment > 50,000 students: including undergraduate, graduate, and professional16 Colleges and > 150 research centers and institutes5,434 faculty members$619 million in research awards in 2010-2011Hundreds of commercial products licensed including Gatorade®

Program: OnMed

• Collaboration between Colleges of Medicine and Education

• Online graduate degree program for clinical health science educators

• Goal: improve the incorporation of educational technology into the educational practices of clinical health science educators

• Program: 36 credits 9 transfer from terminal degree

• Courses were designed to align with ACGME Competencies

Setting: UF Health Science Center

6 Academic Colleges

Setting: UF College of Education

Key College of Education Facts and Figures

Enrollment: 1,863 full-time students at all levelsOnline Enrollment: 4,395Degree Programs: 21Online Degree Programs: 12 degrees offered, all graduate – including 4 online doctoral (EdD) programsFaculty: 106 full-timeResearch Funding: $37.7 million total

Course

• 8 week online course • Moodle• Primarily asynchronous• 3rd course in the program• Critical analysis of academic research was a

primary course objective• Course assignments included weekly discussions

and a brief annotated bibliography

Participants

• N=6• Participants were clinicians and instructors in

the Colleges of Medicine, Pharmacy, and Dentistry.

• College of Medicine Departments:– Pediatrics (N=3)– Ophthalmology (N=1)

Implementation Overview

Instructor Collaboration

• Collaboration began when the course was being designed and continued through to the end of the course and beyond.

• Involvement with the librarian was well documented – in the syllabus and within Moodle.

• The instructor provided input and recommendations for assessing the librarian’s impact.

Moodle Integration

Week Course Module Library Content1 Introduction 1) Forum Introduction

2) Pre-Assessment2 Reading Research 1) RefWorks Handout

2) Refworks Elluminate Session3) What is an Annotated Bibliography4) Critical Analysis LibGuide

3 Research Questions N/A4 Mid-Point 1) Peer Review & Ulrich's

2) Searching Ulrich's Video3) Journal Citation Reports Video

5 Building Your Annotated Bibliography

1) ERIC Demonstration Video2) Why Use MeSH Video3) MeSH Demonstration Video

6 Working on Your Annotated Bibliography

Search Support: 1) Presentation on finding medical education literature 2) General database search techniques

7 Annotated Bibliography

Continued Search Support

8 Issues & Current Research

Post-Assessment

Example

Optional Interaction

Meebo Chat Widget

Discussion Board

Research Questions

How does the presence of an online embedded librarian influence graduate students' experience in an online educational technology research course as defined by:

• Self-efficacy with information literacy and library skills

• Library skill performance

• Quality of research as measured by citation analysis

• Reflections on the experience

Methodology

• Mixed Methods Case Study

• Purpose - Exploratory & Evaluative

Data Collection

• Pre and Post Assessments• Citation analysis• Learner Reflections – 2 open ended reflection

questions• Instructor Interview• Librarian Field Notes

Pre-Post Assessments• 13 items measuring self-

efficacy with information literacy and library skill performance

• Monoi and O’Hanlon & Diaz, 2005

• Delivered online via Zoomerang

• Paired data - analyzed using a paired t-test

Citation Analysis

• Objective Rubric

• Subjective Rubric

• Tunon & Brydges, 2006; 2009

Reflection Questions

• Students were asked to respond anonymously to these questions:

• What role did the librarian (who offered instructional support and assistance) play in your process of completing the annotated bibliography assignment?

• In what ways did your prior experiences in clinical research and literature searching influence your completion of the annotated bibliography assignment and use of the embedded librarian support?

Findings - Overview

• Learners Used the Instructional Materials

• Positive Impact Demonstrated by:

– Increased Self-Efficacy and Performance

–High Quality Annotated Bibliographies

– Themes from the Qualitative Data

Quantitative: Resource Access

Quantitative: Pre-Post & Citation Analysis

Qualitative Themes

1. The nature of the annotated bibliography assignment and critical analysis required in the annotations

2. Use of the library instructional materials

3. Literature search process: search terms and strategies

4. Process of finding useful and relevant articles

Theme 1: Annotated Bibliography

"I approached the annotations as though I

was going to use them to write a manuscript which

helped"

"I was really not getting it until I realized that it is

essentially a summary of what I grasp from a research paper when I am reading it to quote in a manuscript"

Theme 2: Instructional Materials

"The videos from the librarian were helpful to

understand the annotation concept and how it is used."

"I certainly utilized some of the resources provided by the librarian...journal

citation reports and searching Ulrich's video"

Theme 3: Literature Search Process

"my confidence in my ability to conduct sound searches"

"I struggled a bit during the

search""I still have an elementary understanding of how to

decide on terms and combine them in such a way as to produce a useful list of

articles"

Theme 4: Challenges Finding Relevant Articles

"Finding the correct search terms to get the articles that answered

my question"

"Challenges I had with this assignment were

finding publications that were relevant but not

repetitive"

"Literature on it [search topic] has literally exploded so it is hard to sort through what is

relevant and what is not”

Triangulation

• Positive Impact -– Supported by the pre and post-

implementation assessments and learner reflections

• Learner-Content Interactions - – Supported by Moodle access log

data, learner reflections, and instructor interview

Additional Findings

• The importance of faculty-librarian collaboration was another theme that emerged from the data

• Access to the course syllabus and content was critical to the successful design of the content

• The instructor supports library-faculty collaborations and make recommendations for future collaborations

Implications

• Professional Practice

• Administration & Policy

• Curricula

Implications for Librarianship

• Collaboration with faculty

• Selecting courses carefully

• Balancing User Needs & Priorities

• Educating Future Librarians

Policy Implications

• Maximize Impact to Gain ROI:– Reusable Materials– Templates– Scholarship of Teaching and Learning (SoTL)

• Accreditation Standards and Information Literacy

• Supporting Patient Care

Curricular Implications for the HSC Library

• Evidence based medicine

• Importance of measuring long term impact

Directions for Future Research

• Possible Methodologies for Future Studies:

– Case Studies of Other Contexts

– Experimental and Quasi-Experimental Designs to control for other factors

Directions for Future Research

• Potential Research Questions:

– In what stages of the research process could librarians benefit from interacting with a librarian?

– What could the librarian have done differently, especially to increase learner-instructor interactions?

Conclusions

• Online embedded Librarians are the Future!

• Embedded Librarian Implementations:– Should Include customized instructional content

and opportunities for learner-content and learner-instructor “librarian” interactions

– Are context dependent – Evaluation is essential

Thank you!

References Available Upon Request