usability final report

7/30/2019 Usability Final Report

1/25

1

Waer.org Usability Testing

Final Report

Maria Sanfeliu

Nneoma Nwaigwe

Rachel Zaporowski


2/25

2

Abstract

This report outlines usability testing of waer.org. We recruited five participants and gave them

each seven identical tasks to indirectly test different aspects of the website, including design,

navigation, memorability, and efficiency. We also directly requested feedback via

questionnaires to measure the level of user satisfaction. Upon analyzing the results, we detected

both weaknesses and strengths in the current website. Participants demonstrated consistent

difficulty navigating the homepage and locating the search bar while expressing overall sub-

optimal satisfaction. We concluded that waer.org would benefit from renovation and thus

formulated recommendations in accordance with our data.


3/25

3

Table of Contents

Executive Summary 4

Purpose of the Test 5

Objectives and Research Questions 5

Target and Actual Participant Profiles 6

Testing Methods 8

Data and Results 10

Discussion of Results 13

Recommendations 15

Appendices 18


4/25

4

Executive Summary

This report evaluates the usability study conducted for the WAER website. The study was

conducted over a five session period in which five participants with different affiliations to the

radio station were tested. Primary participant characteristics included exposure to the site, level

of technological expertise, and familiarity with waer.org.

Each participant completed an identical set of seven tasks on waer.org and were permitted to

comment openly. Performance and subjective observations were recorded by the usability team

while studying live participant interaction as well as video recordings of participant interaction

via Silverback.

Most Significant Usability Findings Reveal:

A. Participants took advantage of computer functions rather than the website function by using

command + F to locate the end result of a task.

B. Participants generally preferred using the search bar on the bottom of the page to complete a

task as opposed to using the menu for site navigation.

C. Participants were distracted by the overwhelming clutter in design, especially the double-

sided advertisements and layout of the homepage.

D. Participants received multiple error messages when using the search bar to complete a task.

After completing the tasks, participants shared opinions of their experiences with the site by

filling out a post-test questionnaire. They mainly expressed dissatisfaction with aesthetic,

homepage design, and search bar location.

Recommendations:

Create a horizontal menu

Place the search bar at the top of page

Reduce site clutter to improve aesthetic Prioritize updates

Shorten the length of homepage


5/25

5

Purpose of the Test

The team detected numerous usability issues with waer.org upon introduction to the site, ranging

from cluttered design to sub-optimal navigation. We conducted a series of usability tests to

discover the ways in which the interface made it difficult for general visitors to accomplish their

goals. Usability testing generally involves measuring how well test subjects respond in areas of

efficiency, accuracy, recall, and emotional response. Although the tests served as a means to

discover error and areas of improvement through by observing participant navigation, it also took

into account what the site executed well. Therefore, the project intended to:

Analyze data from our observations in order to gauge the level of usability the site

afforded participants: how much time and how many steps were required for people to

complete basic tasks.

Measure the accuracy of the site: how many mistakes participants made.

Assess the level of recall: how much the participants remembered after periods of non-

use.

Evaluate the level of stickiness: how much time participants spent on each task.

Ascertain the degree of user satisfaction: how the participants reacted to tasks and overall

site functions.

In the process of analysis, our final and overarching purpose was to make final recommendations

to the client based on the findings derived from the usability test.

Objectives and Research Questions

Before executing the testing sessions, we organized several objectives in order to create a solid

focal point for the data analysis phase:

Determine the level of site usability by observing participant performance of simple

tasks.

Locate navigation and design errors, which includes errors that lead to an inability tolocate task destinations, excessive use of keystrokes to complete tasks and labeling

ambiguities that prohibit the completion of tasks.

Use collected data to identify issues among different types of users and overall site

pitfalls.

Provide the client with valuable information on website improvement.


6/25

6

We also formulated specific research questions, with the goal of answering them by evaluating

participant performance on our tasks and feedback on our post-test evaluations:

How does the homepage affect user performance?

How do the structure of the menu and location of the search bar affect user navigation?

What obstacles are present with the program schedule?

Do users encounter any problems or obstacles users while using/downloading the listen

live application?

How aesthetically appealing is the website to users? How important is this to them?

Target and Actual Participant Profiles

Target Participants

The original test plan was to recruit five users that would fit into one of two profiles. The first

category, labeled producers, consisted of employees and volunteers at the station. Since the

volunteers are mostly comprised of students, they represented the younger generation of the

participants and we expected them to have a different outlook on using the website. The

members of this group are more likely to be familiar with the site, having to visit it on a regular

basis for information about what is being aired, upcoming events, or just for interests.

The second group, labeled consumers, consisted of the stations listeners within its audience.

According to the NPR profile of 2009, the audience is mostly composed of adult males between

the ages of 45 and 64. The members of this group may or may not be familiar with the site butcould potentially show what a member of WAERs leisure audience is interested in and looking

for.

We hoped the employees and volunteers would represent an expert user profile given by their

amount of exposure to the site and its features, while the listeners would have varying degrees of

site aptitude. We also strived to test users that use the website for different purposes (i.e. work

or leisure) to measure efficiency in different contexts.

Producers

Employees 2

Volunteers 1

Consumers

Listeners 2

_________________________________________________

Total Number of Participants 5


7/25

7

Actual Participants

At the beginning of our recruitment process we reached out to employees and volunteers via

email. Minor setbacks occurred due to inconsistent communication leading us to think for a brief

moment access to employees might not be a possibility as participants in our study. Under the

impression we would not be able to reach our target number of participants, we tested two

volunteers instead of just one like initially planned. We later got in contact with two employees

through the use of social media and a meeting with one of them. As a result we succeeded in the

recruitment of both of employees as participants.

We also reached out to listeners through Facebook and Twitter but did not receive any responses.

This lead to the lack of success in recruitment of participants in the consumer category. We

recruited a participant unaffiliated with the site. The final participant did not listen to WAER or

work for the website or station. Since our previous four participants were already familiar with

the website, we anticipated promising results that might lead to insight as to what a new memberof their audience would think.

Employees 2

Student Volunteers 2

Non-affiliated Student 1

_________________________________________________

Total Number of Participants 5

The chart on the next page documents each users distinct profile. Information was collectedduring the pre-test questionnaire at the beginning of the session.


8/25

8

The compiled user profile reveals that all of our participants were familiar with computers and

the internet, and had at least some familiarity with the format of radio websites. All of them

included communication and music as their main online activities, while four of five mentioned

news and social media. Three stated the search bar as preferred method of navigating any given

website, while the other two preferred using a menu. Employees demonstrated the highest

frequency of visiting waer.org. The most relevant site features among our participants werenews and online listening.

Testing Methods

Data Collection

We set out to collect two types of data: performance and preference. Performance data was

based how much time it took a participant to perform each task, or to realize that they wereunable to complete it, by using a timer and allowing them to verbalize their thoughts. When a

participant insinuated a high level of frustration, we asked if he or she would like to move onto

the next task and made note of task failure. Preference data was to be collected through post-test

questionnaire responses, measuring user satisfaction and perception of the site.


9/25

9

The pre-test questionnaire consisted of multiple choice questions about their affiliation to the

station, previous use of the website and computers, etc. The post-test questionnaire consisted of

questions about their opinions and reactions to the website. Both questionnaires are included in

Appendix D.

Environment and Technology

The waer.org usability tests took place in Huntington Beard Crouse Hall at Syracuse University.

Four of the test were conducted in the Writing Major Lounge using a MacBook Pro. The other

test was given in the Digital Editing Studio where the participant used an iMAC. All of the test

sessions were recorded using Silverback for later analysis.

Our original test plan outlined three different roles for the usability test team. The moderator

was given the responsibility of reading the script to each participant, explaining our reasons

behind the study as well as what was to be expected during the session (see Appendix A). Shewas in charge of executing the test by reading the task scenarios, including handing out the

consent form and questionnaires (see Appendicies B, C, and D). She was the sole communicator

of the testing session, giving assistance to participants if needed. The observer was given the

responsibility of making note of the participants gestures, facial expressions, and comments (see

Appendix E). She was not to intervene in the test itself. The timer used a stopwatch to record

how long each task took for the participants, and also had no involvement in the actual test.

During two of the tests there were two observers in the room, one took notes of the participants

reactions while the other timed the tasks. There was only one observer doing both of these

during the last three tests.

Tasks

The actual test included seven tasks which were read to the participants by the moderator. These

tasks were created to address the different potential problems we identified during our initial

analysis of the website. All participants were given the same set of task scenarios, read aloud by

the moderator. We allowed the testers freedom to progress at their own pace as if they were

using the site in their natural environment. We also offered them the opportunity to ask

questions and express their inability to complete a task if needed. Tasks are listed in Appendix

B.


10/25

10

Data and Results

Performance Data

Quantitative data was collected by timing each participant for each of the seven tasks. This was

intended to measure the efficiency of different aspects the website, as well as identify any areas

of weakness among a specific user profile.

The following spreadsheet documents participant times under the notation

hours:minutes:seconds. Values highlighted in red indicate the time elapsed before a task was

failed*. The subsequent graph illustrates the same data.

*For User 1, Task 3: upon our analysis, we noticed the task was not completed in its entirety.

This graph represents average time needed to either complete or give up on each task:


11/25

11

Trends

Task 1, locating a link on the homepage, posed the most difficulty. Two out of five

participants failed.

Task 3, locating and using the search bar, appears to have posed the second highest

amount of all-around difficulty.

Task 5, locating and using the listen live feature, appears to have the lowest average time

among participants. It was thus the most usable of the site functions.

A general trend of decreasing times with the progression of each test session indicates thesites high level of memorability.

Preference Data

We used the post-test questionnaires as an opportunity to collect user opinions and thus gauge

the level of user satisfaction. They were asked to rate different aspects of the site on a ten-point

scale, with ten representing optimal feedback and one representing poorest level of feedback.


12/25

12

We also asked each participant what he or she would improve about the website. The following

graph represents the frequency each aspect was noted. All five found the homepage

troublesome, while the menu, search bar, and aesthetic were problematic for three different

participants.


13/25

13

Discussion of Results

After careful observation, analysis of the data, and consideration of participant feedback, we

have discovered numerous usability issues with the website. Although the test was mainly

intended to identify problematic features, the study uncovered several strengths as well. Our

findings are presented in accordance with the research questions mentioned earlier.

Q: How does the busy homepage affect user performance?

The design of the homepage proved to be overwhelming to users, confirmed by both

performance and preference data. Its vertical extension requires users to spend much of their

time on the site scrolling up and down. The layout of the links caused users to strain as they

read, searching for the concert link. Advertisements along both sides of the main page were

distracting.

Q: How do the structure of the menu and location of the search bar affect user navigation?

The menu revealed minimal problems among participants, who were all able to navigate the site

with ease using this portal. Upon observing users, it was easily visible and comprehensive. The

location of the search bar at the bottom of the homepage made it difficult to find. As a result, the

excessive scrolling made its use unnecessarily time-consuming.

Q: What obstacles are present with the program schedule?

Performance data on our fourth task scenario, which entailed utilizing the program schedule,

revealed a comparatively low average time for completion. Participants were able to find its link

on the menu and scroll down to the specified program without much hesitation.

Q: Do users encounter any problems or obstacles users while using/downloading the listen

live application?

As this task scenario recorded the lowest average time, user performance revealed this to be the

most usable feature of the site. Users were able to access listen live in more than one way; via

the menu, homepage, and on the side as an advertisement. Multiple means of accessing thisoption made it most efficient.

Q: How aesthetically appealing is the website to users? How important is this to them?


14/25

14

Preference data revealed that site aesthetic was an issues of concern for most participants. Given

the amount of difficulty with the cluttered homepage, an issue based heavily on aesthetic, we

conclude that site appearance is a significant aspect to a website for a majority of users.

We uncovered some strengths of the website, listed as follows:

High level of memorability: on average, users quickly learned their way around the

website

Menu proved to be accessible: users revealed minimal problems navigating with this

portal

Good visibility of several important site features, including listen live and program

schedule

Unintended findings

The Usability Team also discovered and decided to make note of other issues with the site for the

executives convenience:

No events present on the event calendar: when asked to find the link to a concert that

already occurred, a participant went to the calendar and found no events on the calendar,

prolonging her recorded task time.

Search bar results lacked consistency and produced error messages: 2 out of 5

participants received error messages multiple times when they used the search bar to

attempt to find the date of a past concert and the latest final score of the lacrosse game.

Final scores for lacrosse game were often not updated: on separate occasions, 2 out of 5test participants could not complete a task asking to find the final score of the latest

lacrosse game because the latest scores and dates were not updated. The last update was

May 3 while the latest game took place on May 5th.

Participants opted for search shortcuts independent of the site: the first participant made

use of the Mac search function (command + F) rather than the sites search bar after she

discovered an empty calendar of events. She only used site functions to actually

complete a task as a last resort for one task. Also, when asked specifically to use the

search bar to find a task, she found the search bar but did not actually make use of it.

Inconsistency with clickability: when asked to find a news program, a participant clicked

on the news tab and nothing happened. Though the news tab expands into subsections,

other main tabs take participants to their main page as a result of one click.


15/25

15

Recommendations

As a conclusion to our study, the usability team offers our recommendations based previously

presented performance and preference data.

Pursue standard website design: WAERs site design should not abstract too far from

standard site designs. The WAER menu is located at the left side of the page and placed

vertically, blending in with the grey page background. A majority of high-traffic

websites, such as Google, have menus placed horizontally on the top of the page. We

recommend keeping your sites navigation functions close to standard in order to

maximize visibility and enable user confidence.


16/25

16

Relocate the search bar: test results suggest that even employees of the site could not

navigate to the search bar because of its location at the bottom of the page and lack of

proximity to the menu. We recommend that you move the search bar to the top of the

page or closer to the menu.

Reduce site clutter: every extra unit of information in a dialogue competes with therelevant units of information and diminishes their relative visibility. We recommend the

removal of excess textual information. For example, the boxes of links on the homepage

appears with a lengthy description. The average internet user tends to dislike reading

more than necessary, and as a result, may overlook a link they are searching for.

Prioritizing information or grouping information under broader heading should decrease

clutter.


17/25

17

Be mindful of updates: we discovered various tasks could not be completed successfully

due to a lack of site updates. For example, the events calendar was empty and scores for

the most recent lacrosse game was not updated.

General improvement of site aesthetic: preference data revealed that site aesthetic was an

issue of concern for most participants. They expressed frustration with the disorder and

the team also took notice of the extreme length of the homepage. The homepage

contained news-related sections (i.e. NPR Hourly News, Daily News Program) despite

the existence of a news tab on the menu. The same issue was noted for the program

schedule. Also, some tabs on the menu expand into subsections that take the user to a

new page, increasing the potential for error. Compiling all of these options into one page

under one tab will give the users enough options for free navigation without

overwhelming their concept of information volume. We recommend that you reduce the

length of the homepage, removing redundant information, and uniting the menu sub-

options into one tab.


18/25

18

Appendix A - Test Script

Before beginning each test, the following script was presented verbatim to the participant:

Hello, my name is Maria. Im a student here at SU. As the final project for our writing class weare conducting a usability test on the WAER website in order to ascertain what works and what

does not work. The purpose of this test will be to provide the client with valuable information on

how to improve the site. You are one of several participants we will be observing. The test will

consist of a few tasks that we have written down for you. We will need you to let us know once

you have completed each task and feel free to also think aloud as you perform these tasks and

articulate any of your responses to the site. We will not be able to assist you in any of the tasks.

If at any point you feel like you cannot complete a task, let me know and well move on to the

next one.

I will be in the room during the test, explaining the different tasks you will perform. Nneoma

and Rachel will also be in the room observing and taking notes of the test session.

Please make yourself comfortable and take your time. Do not hesitate to share your true feelings

and opinions. We did not have anything to do with the design of the website.

Do you have any questions before we begin?


19/25

19

Appendix B - Task List

How long does it take to find the link to La Vida Boheme, Live In Concert: SXSW 2012 on

the home page?

You wanted to attend the La Vida Boheme, Live In Concert: SXSW 2012 last weekend but

missed it and you want to find out what happened during the concert. Search the home page in

order to find the link for this information.

How long does it take to find the SU Basketball tab on the menu?

You want to find out the score of the SU Basketball teams last game and when their next game

is in the NCAA Tournament. Find this information using the menu.

How long does it take to find the search?

Now you want to find that same information, this time using the search bar.

How long does it take to find a program on the schedule?

You want to catch the latest news as you drive home from work. What time is the All Things

Considered program?

How easy is it to use listen live?

You are stuck in the office and cannot listen to the news in your car like you usually do. Can

you find where to listen to the show on the site?

Can you go back and find the ______ page again? How long does it take?

You forgot the time of the SU Basketball teams next game in the tournament. Go back and find

the time.

Can you access WAERs Twitter page, without leaving the website?

You want to follow WAER on Twitter so you can access their updates from your phone. Find

the WAER Twitter page.


20/25

20

Appendix C - Consent Form

WAER Usability Test- Participant Agreement & Consent Form

Thank you for participating in our usability test of WAER.org. We will be recording the test and

tracking your performance. Only the test moderator and observers will be viewing these

recordings for the sole purpose of collecting data. We will not give out your name or any other

personal information that you give us during the surveys we give you. We will only use your

test results and questionnaire answers for this project.

If you agree to these terms, please read the following information and sign:

I understand that I will be recorded for research purposes during the test. I grant permission to

use these recording for the purpose mentioned above.

Signature: _____________________________________ Date: __________________


21/25

21

Appendix D - Questionnaires

Pre-test Questionnaire

1. How often do you use a computer?

__ Daily

___Often

___Sometimes

___Rarely

___Never

2. How often do you use the Internet?

__ Daily

___Often

___Sometimes

___Rarely

___Never

3. What online activities do you partake in most often? Check all that apply.

__Business

__Communication

__Gaming

__Movies

__Music

__News

__Shopping

__Social media

__Sports

__Other (please specify)

4. Are you an employee or volunteer of WAER?

__Employee

__Volunteer


22/25

22

__Neither

5. How often do you listen to WAER?

__Daily

__Often

__Sometimes

__Rarely

__Never

6. Why do you listen to WAER? Check all that apply.

__Employment reasons

__Leisure__I don't listen to WAER


7. How often do you visit radio websites?

__Daily

__Often

__Sometimes

__Rarely

__Never

8. How often do you visit waer.org?

__Daily

__Often

__Sometimes

__Rarely

__Never

9. For what reason(s) have you visited waer.org? Check all that apply.

__Events

__Local Interest


23/25

23

__News

__Online listening

__I have never visited


10. What features of this website do you use most frequently? Check all that

apply.

__Events

__Local Interest

__News

__Online listening

__I have never visited


11. Do you prefer to use a search bar or a menu?

___Search bar

___Menu

Post-test questionnaire

1 On a scale of 1 to 10, with 1 being poor and 10 being excellent, what was your opinion of

this site overall?

2 On a scale of 1 to 10, with 1 being uncomfortable and 10 being comfortable, how at ease

did you feel about navigating this site?

3 On a scale of 1 to 10, with 1 being inefficient and 10 being efficient, how would you rank

this site?

4 What do you think WAER could do to improve its website?

5 Do you have any other thoughts on WAER.org?


24/25

24

Appendix E - Session Observations

Session 1

-For task 1: went to the calendar and there were no events listed on the calendar.

-She looks confused by the side bar, she furrowed her brows which is a sign of confusion or

hesitation.

-Expresses the fact the site is overwhelming

-Checked the band directory and the calendar

-She took advantage of computer function rather than the website function (command + F)

-Used the find function again, still on the first task.

-I really dont know where it is

-Second task - success

-Third task - she never completed the task, she just found the search bar

-She started using the search bar on the bottom to find task 4 and she found it pretty easily -She used an ad to complete task 5 not the actual menu

-Task 6 is successful

-Task 7 is succesful

Session 2

-Task 1 - successful, a little bit of distraction by the page itself but overall it was successful

-Task 2 - successful


-Task 4- there is no news homepage, the tab was not clickable




Session 3

-Task 3 -is there a search bar, I didnt know there was one- and hes an employee

-He used the search bar and used last years results and then he used the search bar again and got

an error message (Not Found page)

-so the search bar is a fail - It took him a minute and 15 seconds before he gave up.

-Task 6 The score was not updated

Session 4

-hes not very mac savvy

-Task 1 - he couldnt find it on the homepage, hes using the search bar

-he had to bounce around, theres a lot going on with the page

-Task 2 - the scores for the lacrosse games were not updated


25/25

25

-Task 3 - memorability, took a while to find the search bar even though he used it to find the first

task

-Task 4 - hes the only one that used the program schedule on the homepage to find it, everyone

else used the menu or the search bar

-Task 5 - He found the listen live task on the home page,

-Task 6 - He had some trouble using the search menu to complete task 6

-Task 7 - Successful

Session 5

-Problem with the mac, doesnt really know how to scroll,

-He used the search bar, he couldnt find it on the home page, when he tried to find it on the

search bar, he got error messages, 409 page not found but it gave him options, the third time he

tried, the results came up but he didnt see it because it was presented as an ad by google.

-He got an error message multiple times using the search bar.

-Task 2 - successful, happy that he found it-The rest of the tasks were successful.

-this is a terrible website

usability final report

Documents