user testing results october 09. 1. goals and approach 2. task summary 3. results 3.1 virtpresenter...

User Testing ResultsOctober 09

1. Goals and Approach

2. Task Summary

3. Results

3.1 Virtpresenter3.2 YouTube3.3 Google Videos3.4 TED

4. Recommendations

User Testing Results

1. Goals and approach

2. Task Summary

3. Results

3.1 Virtpresenter3.2 YouTube3.3 Google videos3.4 TED

4. Recommendations


2. Task Summary

3. Results


4. Recommendations


Goals and Approach

Goal: Evaluate the usability of the following media players:

Virtpresenter

http://video.lernfunk.de/watch?seminar=algorithmen08&lecture

=algorithmen08_2008_11_10_14_18__131_173_10_32&related=studipuos&token

=fb2eb23d9d20d662d508f95cb180eb98#app=e43a&81f8-selectedIndex=1

Youtube

http://www.youtube.com/watch?v=43DuLcBFxoY&cc=1

Google videos

http://video.google.com/videoplay?docid=-7011751420967002369&ei=XeDBSqSIL83F-AaB5ZDgBg&q=

tedtalks

TEDhttp://www.ted.com/talks/lang/eng/hans_rosling_shows_the_best_stats_you_ve_ever_seen.html

http://video.lernfunk.de/watch?seminar=algorithmen08&lecture=algorithmen08_2008_11_10_14_18__131_173_10_32&related=studipuos&token=fb2eb23d9d20d662d508f95cb180eb98#app=e43a&81f8-selectedIndex=1
























http://video.google.com/videoplay?docid=-7011751420967002369&ei=XeDBSqSIL83F-AaB5ZDgBg&q=tedtalks












http://www.ted.com/talks/lang/eng/hans_rosling_shows_the_best_stats_you_ve_ever_seen.html










Goals and Approach

Approach: Systematically observe potential users of the examined media players performing representative, determinate tasks.

Goal: Evaluate the effectiveness, efficiency, and satisfaction of Virtpresenter, Google Videos, YouTube, and TED. The participants were given a series of tasks to complete with each player and were asked follow up questions directed towards understanding:

whether the structure of the players are easy to understand

whether users can find the necessary information to complete their tasks

whether the user can navigate the content easily

Participants were comprised of 5 students from The Open University of Catalonia (UOC).Each participant completed the tasks individually in 40-50 minute sessions.Study was conducted on October 3rd at the University Autónoma.

Goals and Approach

Name Profile

User 1 Student of UOC - Psychology -

User 2 Student of UOC - Education -

User 3 Student of UOC - Computer Science -

User 4 Student of UOC - Computer Science -

User 5 Student of UOC - Nursing -


2. Task Summary

3. Results


4. Recommendations


2. Task Summary

3. Results


4. Recommendations


Task Summary

Each user was given the a number of scenarios with corresponding tasks.

While completing the tasks, the participants were observed closely and each task was attributed either a success or failure score.

Next we present a table displaying the tasks and their corresponding results.

Task Summary

Task Success Ratios [Virtpresenter]

Scenario Task User %You have registered for a class and found out that all the lectures are available online. You have opened the link to the lecture viewer with the intention of watching the first day of class.

Begin watching the first lecture and watch for 15 seconds

5/5 100%You just received a phone call that you need to take.

Pause the lecture so you can take your call. 5/5 100%

You will need to meet the person you were talking to in an hour meaning you need to leave in 30 minutes. You are curious whether you will be able to finish the lecture before you need to leave.

Find out how much of the lecture you have watched and how much is remaining.

4/5 80%Now that you’ve realize you won’t have enough time to watch the entire lecture, you decide to skip to the __________ section which you think is the most important. You’ll watch the rest when you get home tonight.

Begin watching the __________ section of the lecture.

2/5 40%Your roommate is listening to his music loudly and you are having trouble concentrating. You would like to turn the volume up and make the slides bigger so you can focus on the lecture.

Turn up the volume and Make the video of the professor larger and continue watching. [no applications]

[no aplications] [no aplications]

Virtpresenter

Task Summary

Task Success Ratios [YouTube]

Scenario Task User %You cannot hear (or the video is in the wrong language) and you would like captions.

Turn on the captions so you can read what is being said. 3/5 60%

You received a phone call and would like to pause the video. Pause and play the video.5/5 100%

The video is too small and you cannot see any details. Make the video larger so it takes up the entire screen. 5/5 100%

Your roommate is playing their music too loudly and you can’t hear the video.

Turn up the volume to its maximum level.5/5 100%

You have already seen the first half of the video and would like to watch the second half.

Jump to the second half of the video and watch it from there. 5/5 100%

Youtube

Task Summary

Task Success Ratios [Google Videos]



You received a phone call and would like to pause the video. Pause and play the video.5/5 100%

The video is too small and you cannot see any details. Make the video larger so it takes up the entire screen. 5/5 100%





Google videos

Task Summary

Task Success Ratios [TED]



You received a phone call and would like to pause the video. Pause and play the video. 5/5 100%The video is too small and you cannot see any details. Make the video larger so it takes up the entire screen. 5/5 100%Your roommate is playing their music too loudly and you can’t hear the video.




Google videos


2. Task Summary

3. Results


4. Recommendations


2. Task Summary

3. Results


4. Recommendations


Results (Virtpresenter)

Scenario 1

You have registered for a class and found out that all the lectures are available online. You have opened the link to the

lecture viewer with the intention of watching the first day of class.

Task 1.1 – Begin watching the first lecture and watch for 15 seconds. 100% 0%

Had Little to No Difficulty (5 users):

• All users clicked the large play button found in the video screen.

• None of the participants noticed the play button next to the scrubber.

• One user commented that he would like to download the video to watch on his computer at another time but he couldn't find this button (doesn’t exist).

“I want to download the video to watch whenever and wherever I want”. User 4.

Watch video

https://wiki.opencastproject.org/confluence/download/attachments/8552658/virtpresenter+1.1.mp4


Scenario 2

You just received a phone call that you need to take.

Task 2.1 – Pause the lecture so you can take your call. 100% 0%


• 3 participants clicked the larger pause button in the video section [button not visible until video section is hovered over]

• 2 participants clicked the smaller pause button next to the scrubber. They didn’t notice the large button in the video section]

“If the icons would be under the video, like in YouTube, everybody would se them”. User 4.

Watch video

https://wiki.opencastproject.org/confluence/download/attachments/8552658/virtpresenter+2.1+.mp4


Scenario 3

You will need to meet the person you were talking to in an hour meaning you need to leave in 30 minutes. You are curious whether you will be able to finish the lecture before you need to leave.

Task 3.1 – Find out how much of the lecture you have watched and how much is remaining. 80% 20%


• Participants found the time quickly and without problems. When they tried to move the timeline, they didn't distinguish the blue circle from the orange square.

• They didn’t understand the concept of “bookmarks” and they didn’t understand the two oranges squares.

• They believed the orange square was the playhead because it is positioned on top of the scrubber [leading them to think they could move it].

• Other students believed this was a delimiter and the video would stop at the last orange box.

• They expected to be able to type the numbers into the timecode.

Had Difficulty (1 user):

• Didn’t see the numbers and didn’t understand the system of blue circles and oranges squares.

Watch video

https://wiki.opencastproject.org/confluence/download/attachments/8552658/virtpresenter+3.1.mp4


Scenario 4

Now that you’ve realized you won’t have enough time to watch the entire lecture, you decide to skip to the __________ section which you think is the most important. You’ll watch the rest when you get home tonight.

Task 4.1 – Begin watching the __________ section of the lecture. 40% 60%


• They had a hard time finding where to look.

• Two participants tried using the forward and backward buttons found above the scrubber but became confused.

• One participant tried dragging the scrubber but it took a while for them to find the playhead; they expected to see it larger and at the top of the scrubber.

Had Difficulty (3 users):

• They did not see the forward and back buttons.

• They did not see the accordion under the video.

• No participants understood the chapters located in the scrubber and did not realize that it was possible to browse through them.

“Maybe if the accordion were in a different color I would notice it. I prefer to move with the scrubber”. User 4


Scenario 5

Your roommate is listening to his music loudly and you are having trouble concentrating. You would like to turn the volume up and make the slides bigger so you can focus on the lecture.

Task 5.1 – Turn up the volume and make the video of the professor larger and continue watching. [no applications]


• They expected to find a button like on other players that would open full screen. They thought this full screen was not big.

• Two users found the icons plus and minus on the video, but they associated them with volume. When clicking on them, didn’t understand the functionality and when they reached the big video screen, they did not know how to get back.

• They expected to find a volume icon as on other players.

“This has confused me. I thought that it was the volume. I expect an icon for “full screen" that makes the video big ”. User 1.

Watch video

https://wiki.opencastproject.org/confluence/download/attachments/8552658/Virtpresenter+5.1.mp4


2. Summary of tasks and dysfunctions detected

3. Task’s Details


4. Improvement opportunities


2. Summary of tasks and dysfunctions detected

3. Task’s Details


4. Improvement opportunities


Results (Youtube)

Scenario 1

You cannot hear (or the video is in the wrong language) and you would like captions.

Task 1.1 – Turn on the captions so you can read what is being said. 60% 40%


• Had difficulty finding it and needed a lot of time to find the button.

• All of them said that the captions icon wasn’t intuitive and the reason they spent lot of time to find it.

Had Difficulty (2 users):

• One of them found the icon when she tried to find the full screen button. In general, she didn't understand the buttons.

• The other user couldn’t find the icon.

“The icon and interaction is not intuitive”. User 2.

Watch video

https://wiki.opencastproject.org/confluence/download/attachments/8552658/Youtube+1.1.mp4

Results (Youtube)

Scenario 2

You received a phone call and would like to pause the video.

Task 2.1 – Pause and play the video. 100% 0%


• Three participants used the pause button in the scrubber.

• Two participants clicked the video to pause the content.

Watch video

https://wiki.opencastproject.org/confluence/download/attachments/8552658/youtube+2.1.mp4

Results (Youtube)

Scenario 3

The video is too small and you cannot see any details.

Task 3.1 – Make the video larger so it takes up the entire screen. 100% 0%


• They found the full screen button very quickly.

• When they needed to close the window, they couldn’t find the icon. They associated the icon (X) with the action of closing the entire window; they assumed it would close the entire browser.

• One user closed the full screen by double-clicking on the video.

“This icon is for closing youtube”. User 3.

Watch video YouTube

(Close Full Screen)

YouTube

(Close Full Screen Hover)

https://wiki.opencastproject.org/confluence/download/attachments/8552658/youtube+3.1.mp4

Results (YouTube)

Scenario 4


Task 4.1 – Turn up the volume to its maximum level. 100% 0%


• All users knew how to use the volume icon.

• They knew if they clicked the button, the volume would mute.

• They know that dragging would raise and lower the volume.

Watch video

https://wiki.opencastproject.org/confluence/download/attachments/8552658/Youtube+4.1.mp4

Results (YouTube)

Scenario 5


Task 5.1 – Jump to the second half of the video and watch it from there. 100% 0%


• They found it dragging the scrubber to the X point.

• Two students tried writing in the “timecard box”

“I like that I can see the time dragging in the scrubber”. User 4.

Watch video

https://wiki.opencastproject.org/confluence/download/attachments/8552658/Youtube++5.1.mp4


2. Task Summary

3. Results


4. Recommendations


2. Task Summary

3. Results


4. Recommendations


Results (Google videos)

Scenario 1




• It wasn’t hard to find it.

• All users said this icon was more intuitive than the one found in YouTube.

“I prefer this icon more than the other one”. User 2.

Watch video

Google YouTube


Scenario 2




• All users used the icons in the scrubber.

• Some users paused the media by clicking the video.

Watch video


Scenario 3




• All participants found the button very quickly.

• All the users said that this icon was easier to understand than the one in YouTube.

• When trying to exit full screen, three participants hit the Escape key but were asked to only use the interface.

Google YouTube YouTube Hover


Scenario 4




• All the users knew how to use the volume icon.

• They knew clicking would mute the volume.

• They know that dragging will raise or lower the volume.

• They prefer the YouTube icon more than this one.

Watch videoGoogle YouTube


Scenario 5




• Three found it by dragging the scrubber to the X point.

• Two participants used the video preview images


2. Task Summary

3. Results


4. Recommendations


2. Task Summary

3. Results


4. Recommendations


Results (TED)

Scenario 1




• For three of them, this was more difficult than both YouTube and Google Videos because they expected to have all the icons in the same place.

• They liked it because it’s not an icon but they still wanted the Google Videos icon for captions and this one for subtitles.

“The icon and interaction is not intuitive”. User 2.

Watch video

YouTubeTED

Results (TED)

Scenario 2




• All participants used the icons in the scrubber and found it very quickly.

Watch video

Results (TED)

Scenario 3




• Participants had some difficulty finding the full screen button.

• All of them said they expect to have this button at the same level than the others, like YouTube.

“Oh! It´s here! I didn’t expect to find it here!”. User 3.

Watch video

TED YouTube

Results (TED)

Scenario 4




• All participants knew how to use the volume button.

• They thought it wasn’t easy to adjust the volume.

“I want to play and pause the video by clicking the video screen”. User 3.

Results (TED)

Scenario 5




• All participants dragged the scrubber to the X point.

• One participant tried to type in the timecode.

Watch video


2. Task Summary

3. Results


4. Recommendations


2. Task Summary

3. Results


4. Recommendations


Recommendations

Based in the user test we present the following recommendations for improvement that some should be made to resolve of the most severe problems that we found during the usability test carried out on:

Recommendations

• Play and Pause

-Location: the majority of participants clicked the play button found in the video section and the pause button found in the scrubber (aka controls bar). The Matterhorn media player should offer both options.

- Video Section Button Visibility: users are reluctant to have elements that obstruct the video content while they are watching. It is not recommend to show buttons in this section unless their cursor is in the center. Only have the Pause button visible when the cursor is directly over it.

• Scrubber

- Participants did not understand the difference between the playhead and the bookmark sliders. Consider separating this functionality.

- Some participants wanted to be able to specify the timecode by typing it in. Include editable timecode box so users may jump to specific sections of the content.

- The users liked the dynamic tooltip that appears while dragging the scrubber. This made it easy for them to identify the part of the lecture to which they are navigating. Include a timecode tooltip while playhead is dragged across the scrubber.

Recommendations

• Full screen

- Some participants expected the screen-size buttons to adjust the volume. They expected to find a full screen button on the lower right.

- In Virtpresenter, some participants expected the screen-size buttons to adjust the volume. Follow common conventions of full screen button such as YouTube and Google Videos. - Participants expected to have the functionality of making the video take up the entire screen with no controls visible. Include a full screen implementation like YouTube.

• Volume

- Users preferred collapsible volume controls (YouTube) over slider (Google Videos). Utilize collapsible control convention for volume.

• Slider

- Users were confused by the chapter sections in the scrubber. Consider a different implementation or inclusion of color to make this functionality more apparent.

- Users were confused and unsure what to expect by the playback navigation arrows. Group and place controls appropriately to ensure transparency.

Recommendations

• Captions

- Of the two versions of captions provided, users prefer the first [A] which overlaid text on the video content. Have captions overlay video content and include a semitransparent character background to improve contrast and therefore readability.

[A] Inside de video screen.http://www.strongis.com/testing/Test2/

[B] Outside de video screen.http://www.strongis.com/testing/Test1/

YouTube TED

http://www.strongis.com/testing/Test2/

http://www.strongis.com/testing/Test1/

Recommendations

Other Suggestions

- Students want the opportunity to download the video and view it offline.

- Students do not want the video to begin playing on page load. Do not have video play automatically when the page loads.

- Control buttons should be kept together and not separated like the full screen button and subtitles button in TED. Organize controls and keep them together.

- They do not understand the concept of bookmarking. Consider redesigning this interaction.

- Participants did not search inside the accordion. Reconsider placement and design of accordion.

user testing results october 09. 1. goals and approach 2. task summary 3. results 3.1 virtpresenter...

Documents