TransTrackerPilot Usability Study

Team:Drew Bregel - Development, Data Analysis

Marianne Goldin - PM, UI Tester, Data Gathering, Presenter

Joel Shapiro - Tasks, Interactions

Joe Woo - Development, UI Tester

Our Tool

• A mobile phone application that goes above and beyond Google Maps

• Build on Metro TripPlanner, OneBusAway, Google Maps transit trip planning

• A more visual/map-heavy interface (less text input), shows the user the context of their environment

• Importantly: the app predicts what the user will do next, and knows what the user does frequently!

Introduction to the Experiment

1. After redesigning our program since the first usability test, we wanted to see how users interacted with TransTracker and expected it to function.

2. We video-recorded the tests with the users and collected data based on their performance and opinions on the program and how they interacted with it.

Method

• 5 Volunteer Participants• Application running on an emulator on a

PC Laptop• Private room in the CSE Basement• 2 testers: One to conduct the test, One

to record the test, bring in the volunteers, and troubleshoot the app

• Payment in form of a snack (pop, chips)

Test Procedure

1. Recruit volunteer2. Consent form3. Introduction to our application (avoid key-words used in UI,

explain “think out loud”)

4. Participant reads task scenario out loud and does the task

5. Time each task (each user must complete their tasks)

6. Videotape each task7. Demographic questions8. Debriefing comments9. Compensation: a snack from the student lounge

Participants - Demographics

• 5 participants– Median age = 22– 3 males, 2 females– All upperclassmen, CSE majors

• 3 of 5 used mobile applications• All used web-based transit applications

– Note: The 2 who did not use mobile apps, used web-based apps the same or more (our potential customers!)

• All used the bus at least once a week (average= 6.8 times/week)

Test Measures

• Dependent Variables (for each task)– Time to complete incident– # Errors – # Times user goes to wrong screen or

scrolls to wrong area

• Critical Incident Logs (for each task)– Both positive and negative– Transcribed from video

Tasks

• We defined a list of task scenarios that would make use of the principal features of our programs and gave users context for the tasks.

Task 1: Predicted Location (Easy)

• Using our app’s predictive feature, take a trip to your most frequent destination (“Home”)

• The user can assume that one of the predicted routes is accurate since they have taken it before

Task 1 Screens

Task 1 Screens: Detail

Task 1: Looking for…

• If user tried to select any tabs that were not needed to complete the task.

• If the user attempted to scroll down pages when unnecessary.

• Are users unclear on where click to take a trip?• Are users aware that they can click on a

destination to take a trip?• Is it obvious that the Home screen is the default

screen in the program?

Task 2: Saved Location (Medium)

• Find a trip you’ve taken before, and take it now

• The user can assume that the destination that they would like to go to is listed in the “Saved” page

Task 2 Screens

Task 2: Looking for…

• Does the user [mistakenly] select the New tab for this task?

• Does the user understand the concept of Saved trips?

• Does the user understand that they can re-take a trip that they’ve taken?

• Does the user understand expect that the destinations and trip times are clickable?

Task 3: New Destination (Hard)

• User must enter anew trip into the program through the “New” page.

• The user is to make their new trip by searching for it, then taking the trip.

• The user has not taken the trip before

Task 3 Screens

Task 3: Looking for…

• Does the user understand the idea of a New trip?

• Does the user understand how to select a destination from the search results?

• Does the user understand what the map represents?

• Does the user expect that the destinations and trip times are clickable?

Study Results

• Collected both in real-time during the test…– Task time– # Errors– # Wrong Screens

• And retroactively, through reviewing video of the tests– Critical results– Verification of # Errors / # Wrong Screens– Notes and chatter

Completion Time

Participant 1

Participant 2

Participant 3

Participant 4

Participant 5 Average Median Best Worst

Seconds to complete task 1 110 11 35 110 95 72.2 95 11 110 Seconds to complete task 2 31 11 18 12 55 25.4 18 11 55 Seconds to complete task 3 39 20 17 22 61 31.8 22 17 61 Total Time for all Tasks 180 42 70 144 211 129.4 135 39 226

Completion time

Errors Participant 1

Participant 2

Participant 3

Participant 4

Participant 5

Average

Median

Best

Worst

# Errors in task 1 5 0 1 4 1 2.2 1 0 5 # Errors in task 2 0 0 0 0 1 0.2 0 0 1 # Errors in task 3 0 0 1 0 2 0.6 0 0 2 Total Errors for all Tasks 5 0 2 4 4 3 1 0 8

# Errors

Irrelev ant Screen

Participant 1

Participant 2

Participant 3

Participant 4

Participant 5 Average Median Best Worst

# Times irrelevant screen in task 1 2 0 1 1 2 1.2 1 0 2

# Times irrelevant screen in task 2 0 0 0 0 2 0.4 0 0 2

# Times irrelevant screen in task 3 0 0 0 0 0 0 0 0 0 Total I rrelevant Screens for all Tasks 2 0 1 1 4 1.6 1 0 4

# Times on an Irrelevant Screen

Critical Incident Count

Participant 1

Participant 2

Participant 3

Participant 4

Participant 5 Average Median Least Most

Task 1 Positive 0 0 0 2 1 0.6 0 0 2 Task 1 Negative 5 0 3 4 5 3.4 4 0 5 Task 2 Positive 2 0 1 1 3 1.4 1 0 3 Task 2 Negative 0 0 0 1 5 1.2 0 0 5 Task 3 Positive 1 1 0 3 1 1.2 1 0 3 Task 3 Negative 0 1 1 0 7 1.8 1 0 7 Neutral/ Other Incidents 1 0 0 1 5 1.4 1 0 5 Total Critical Incidents for all Tasks 9 2 5 12 27 11 9 2 27

Critical Incident Count

QuickTime™ and a decompressor

are needed to see this picture.

Sample Video of Task 3

• http://www.youtube.com/watch?v=szcLDylxhtM

Major Themes in Negative Critical Incidents

Task 1: • List comprehension• Map interactions• Task completionTask 2: • Prototype fidelityTask 3:• Wizard of Oz• List comprehension• Task completion

Task 1: List Comprehension

• Affordances of the items on the list in the home screen

• Context of the list - why is it important?

Task 2: Prototype Fidelity

• Not all items that should be clickable, are actually clickable

Task 3: Wizard of Oz

• Lots of false positives--it would be great if our app could read the user’s mind

Tasks 1-3: Task Completion

• Since users did not understand the spirit of the application, they are not sure when they’ve gotten all the info they can

• Users not sure what the “end point” of the task is

Recommendations for Design Changes

• Differentiate between clickable and non-clickable items

• Rename tabs• Replace scroll option with a screen expansion option• Clarify how the predicted locations on the Home

screen are sorted

Recommendations for Design Changes

(continued)

• Tighter integration with Google Maps cues• Destinations and search results need more

detailed/contextural info• Add clear exits to all pages that lack them• Increase use of symbols and images through

the programs• Make current time visible and obvious

Summary

• The questions that we asked our users they gave us a great set of data; almost exactly the kind of results that we were hoping for

• Our script accurately reflected the tasks that we wanted our users to accomplish and led them on the paths that we wanted to find UX errors that we wanted to correct

• Our program uses a small enough set of pages that it was easy to see overarching issues in the program