usability report cover sheet - lib.umich.edu · usability assessment is a part of the iterative...

VuFind Task Force • Login Guerilla Test

University of Michigan | MLibrary | [email protected] This work is available under a Creative Commons Attribution Non-Commercial

3.0 License. http://creativecommons.org/licenses/by-nc/3.0/us/.

© The Regents of the University of Michigan, 2008.

UsabilityReportCoverSheet

Project • Overall Project: VuFind • Project Title: Login Guerilla Test

Committee & Members

• Usability Group: Suzanne Chapman (chair), Shevon Desai, Kat Hagedorn, Julie Piacentine, Ken Varnum

• Usability Task Force: Shevon Desai (co-convener), Kat Hagedorn (co-convener), Ken Varnum (stakeholder), Scott Ash (stakeholder), Marisa L. Conte, David Fulmer, Beth Strickland

Report Info

• Report Author(s): Shevon Desai and David Fulmer • Contact Information: [email protected] • Report Date: November 2008

Online Report Location

• http://www.lib.umich.edu/usability/projects/ProjectReports/Login_guerilla_final.pdf

• http://www.lib.umich.edu/usability/projects/ProjectReports/Login_prelim_findings.pdf


University of Michigan | MLibrary | [email protected] 1

Usability Report

Summary: Test Design and Methodology ...................................................................................... 1

Test Results..................................................................................................................................... 3

Analysis........................................................................................................................................... 4

Recommendations........................................................................................................................... 5

Appendices...................................................................................................................................... 6

Appendix A: Test Script ............................................................................................................. 6

Appendix B: Screenshots Used for Testing ................................................................................ 9

Summary:TestDesignandMethodology

VuFind is the open-source, next generation discovery tool that will soon be used to provide access to the MLibrary catalog. VuFind is not currently slated to replace Mirlyn as the main library catalog, but it will work in conjunction with Mirlyn.

Goalsforguerillatest*• Determine how users perceive having to login when using catalog • Determine at which access point(s) it is best to force a login decision (catalog home page,

at search results screen, or at item record screen) • Determine whether an embedded (VuFind embedded within the MLibrary template)

interface vs. standalone (VuFind not embedded within the MLibrary template) interface makes a difference in regard to login decision

Methodology:guerillatest• Two task force members (David Fulmer and Shevon Desai) approached users in various

public areas (2nd floor of Graduate Library, Michigan Union, Angell Hall “fishbowl,” Taubman Medical Library, Glass House Café and café in Public Health building)

• Presented users with paper screenshots and asked a series of questions • Test took approximately 7-10 minutes each • Users offered candy as small incentive

* This report describes a usability assessment performed by the MLibrary Usability Group and Task Force. Usability assessment is a part of the iterative design process used to develop new systems and services at MLibrary. This report describes one test in a series of tests performed at one point in the iterative design process. Sample sizes are small and findings serve only as clues to help guide decisions. Implementation of any recommendations should take these limitations into account.



TestDesign

• Each participant was asked whether he/she would login or continue without logging in at 3 different points when using the catalog. Each was presented with a separate “scenario”

o Catalog home page (“this is the library catalog home page”) o Search results screen (“you searched for some books in the catalog”) o Item record screen (“someone has sent you a link to a book in the catalog”)

• Users looked at paper screenshots when presented with each question • Each user was presented with screenshots in which the VuFind interface was either

embedded in the MLibrary template or the VuFind interface was presented separately with a different template (standalone)

• For each screenshot, participants were shown dialog boxes forcing a login decision (see Figures 1 and 2 for examples) before they could continue

• At end of test, participants were asked at which of the three access points they preferred to be presented with the choice to login or not

• See Appendix A for test script • See Appendix B for all screenshots

Figure 1 Screenshot D: Embedded catalog home page



TestResults• 16 people total participated in the guerilla tests • 8 viewed the embedded pages • 8 viewed the standalone pages Participants who viewed embedded pages (8 total) Q1 (When presented with the catalog homepage, “would you choose login or continue without login”)

• 5 of 8 would login, 3 of 8 continue without login

Q3 (When presented with search results in the catalog, “would you choose login or continue without login”)

• 5 of 8 login, 3 of 8 continue without login

Q5 (When presented with an item record, “would you choose login or continue without login”) • 3 of 8 login, 5 of 8 continue without login

Q9 (“At what point would you prefer to log in among the three pages”)

• 4 of 8 prefer login at beginning (catalog homepage)

Figure 2 Screenshot E: Standalone catalog home page



• 2 of 8 indicated they preferred to login only if they had to in order to get the information needed (participants #7 and #13)

• Rest varied: 1 said before or after completing a search; 1 person (#11) chose none of 3 options presented - he approached test as if he were viewing the pages from within the library itself, and generally did not want to be asked to login for "each and every thing."

Participants who viewed standalone pages (8 total) Q1 (When presented with the catalog homepage, “would you choose login or continue without login”)

• 7 login, 1 continue without login Q3 (When presented with search results in the catalog, “would you choose login or continue without login”)

• 1 login, 7 continue without login Q5 (When presented with an item record, “would you choose login or continue without login”)

• 1 login, 7 continue without login Q9 (“At what point would you prefer to log in among the three pages”)

• 6 prefer login at beginning (catalog homepage) • 1 preferred only to be asked to login when had to in order to use a feature/service • 1 person was not affiliated with UM, therefore not asked this question

Analysis• Majority of participants for both groups (5 of 8 embedded, and 7 of 8 standalone) would

choose to login when presented with login choice at catalog home page • When presented with login option at other access points, results varied • Majority of standalone viewers (7 of 8 for each) chose not to login at the search results

nor item record page • Results not as clear cut for embedded viewers (5 of 8 would login at the search results,

and 3 of 8 at the item record page) • Some people will always login whenever they are presented with the option • Some people prefer to login only at the point at which they can no longer get what they

need without logging in • There is consensus that being asked to login at “beginning” (i.e., the catalog home page)

is not perceived as an intrusion • User expectations of the catalog and how it works make a difference • If user thought he/she would have access to needed services they are willing to login at

whatever point. However, if he/she perceives that needed information is readily available regardless of login status, will continue without logging in

• Scenarios we presented ("you did a search in the catalog and are now looking at results," and "someone sent you a link to a particular book in the catalog") do not imply that users will need a service for which one would eventually have to login. Some users assumed they might need login for “full-text" or to save results. They assumed they would have to login at some point, regardless



Recommendations• Present login option at catalog home page • Do not present login option again until user is forced to login to get desired service (for

example: saving search results, placing a hold on an item) • Do not present login option to users viewing individual catalog records



Appendices

AppendixA:TestScript

Greet the participant Hi, my name is_________. Would you be willing to participate in a brief study about one of the library’s services? It will take about five to ten minutes of your time. (Yes or No)

Pre-test procedure

Thank you for participating in this study. I do want you to know a couple of things before we begin:

• I want to emphasize that we are not testing you or your abilities – we are testing the product. We are here to learn from your experience.

• You should know you are not obligated to take this test, but we hope you will, to help us improve our library’s resources.

We need you to sign a consent form for our IRB (behavioral study) records. Your answers will be anonymous and we need the information only for analysis purpose.

Thanks! Do you have any questions for me before we begin?

Begin Test Part A: What is your status?

• Faculty • Staff • Graduate Student • Undergraduate Student • Visitor

Do you have a UM login (if not sure, show UM authentication page)?

Part B: Half of the users see screenshot D (embedded). The other half of the users see screenshot E (standalone). Stay consistent in showing embedded vs. standalone for Parts C and D.



This is the library catalog homepage which gives you the option to login or not by default. You will need to select a login option before you can continue.

• Q1: Would you choose "Login" or "Continue without login"?

• Q2: Why did you decide to choose that option? Why didn't you choose the other option?

Part C: Show the user the home page (screenshot F). You've done a search from this page, and now you're looking at results from the library catalog. You will need to select a login option before you can continue. Show half the users screenshot G (embedded). Show the other half of the users screenshot I (standalone).



Part D: Show half the users screenshot H (embedded). Show the other half of the users screenshot J (standalone). Someone is recommending that you look at a book in the library, and they've sent you a link to the catalog. You will need to select a login option before you can continue.



Part E: You've seen three options for logging in: when first going to the catalog, after completing a search, and when viewing a particular book in the catalog. Show the user screenshots D, G and H OR screenshots E, I and J. Q7:

• If user has selected login 2 or more times: What benefit do you see in logging in? • If user has selected login 0 or 1 times: What features might make logging in beneficial

for you?



Part F (do not ask visitors): Logging in allows you to place a hold on a book, save a search, tag an item, among other options.

• Q8: Does that change what you think about logging in? • Q9: At what point would you prefer to log in among the three pages? (have them choose

one)

Thank you for participating in this study. We appreciate your help! Feel free to take a piece of candy.



AppendixB:ScreenshotsUsedforTestingFigure 3 UM authentication page



Figure 4 Screenshot D: Embedded library catalog home page



Figure 5 Screenshot E: Standalone library catalog home page



Figure 6 Screenshot F: New library homepage



Figure 7 Screenshot G: Embedded search results from the library catalog



Figure 8 Screenshot H: Embedded link to a book in the catalog



Figure 9 Screenshot I: Standalone search results from the library catalog



Figure 10 Screenshot J: Standalone link to a book in the catalog

usability report cover sheet - lib.umich.edu · usability assessment is a part of the iterative...

Documents