usability report - dialogdesign · web viewformal usability testing was chosen as the method of...

HotmailUSABILITY REPORT

Comparative Usability Evaluation – CUE-2Team F

November 1998

SummaryOverall, users had no serious problems using the tool. Users referred favorably about it.

Users found Hotmail through the hotmail.com URL. The initial interest to Hotmail was high. First-time users read the Visitors and Info sections information carefully.

There were some minor attitude and usability concerns during the initial use of the application. For example, users thought that user information was too "gimmicky," users did not like to enter personal data when registering, users interpreted the Password hint incorrectly, users expected the Enter key to work for moving between the login fields. The Forgot your password? feature was appreciated by users.

Users found all main features of the mail tool - Inbox, Composer, Filters, Folders, etc. - easily. There were no major usability problems. However a number of minor problems were observed. For example: The content of Related was different on different pages; users did not understand why. The difference between Dictionary and Thesaurus may not be obvious to all users. Users did not understand the term Nicknames. POP mail caused confusion. Filters descriptions talked primarily about trashing junk mail. Controls for setting up a filter seemed awkward.

There were some technical problems, for example, the site did not provide a means for alternative navigation, the speed of loading was insufficient, and the application crashed too often.

Comparative Usability Evaluation, CUE-2 Team F

BackgroundThe purpose of this test was to test navigation and content of the current version of the Hotmail web-based mail tool. Specific areas and questions of interest were identified by the development team prior to the test. Please refer to Appendix 1 for the Test Scenario.

The tests were conducted November 1-30, 1998. The duration of testing relates to the fact that I did not have a time period that I could devote to the Hotmail testing alone. Five participants tested the tool. Please refer to Appendix 3 for participants profiles.

Formal usability testing was chosen as the method of testing for the following reasons: This is one of the most efficient methods (in terms of the time and effort vs. results

ratio). Formal usability tests are the test most often conducted in our lab and I considered

it useful that we not change our practices in the comparison study. The chosen method was not 100% efficient due to the nature of the tool and the questions posed by the development team. If I had an opportunity to renegotiate test questions and methods, I would suggest combining formal usability testing with other usability methods - surveys of a large audience of Hotmail users and/or user diaries. I did not have this option. I commented on the deficiencies and possible ways to gather more information where appropriate in this report.


Findings and Recommendations

Home page

Access

Users found Hotmail through the hotmail.com URL. Users preferred the Hotmail page to the MSN page. Users could not find Hotmail on the MSN page.

All five testers, when asked to find Hotmail, accessed it by typing Hotmail or hotmail.com, or www.hotmail.com in the URL field. Two of the testers commented that they remembered hearing about Hotmail. One user said that he received email from a friend and saw the URL there.For the last three tests, I made the MSN page the default page - the browser came up with it. When on that page, I gave users a standard instruction to find the Hotmail tool. None of them even looked at the MSN page. Later in the test, I asked two users to look at the MSN page again and see if Hotmail is available there. They did not find it. One user commented that what's available on the MSN page does not have any resemblance to Hotmail, and the color combination is totally different too. Another user, when looking for Hotmail on the MSN page, tried clicking the word "email" there; it was on the blue background but was not a link, so she lost interest. All three users communicated that the MSN page looks very busy and really disorganized.Recommendation: Keep the Hotmail page as the main location and access page. Redesign the MSN page if accessible. Note: Formal usability testing may not be an ideal method for clarifying this question. A method that can prove helpful would be to conduct a survey with a representative sample of users. (See my discussion of test organization in Appendix 3 for more details.)

Login fields

Users expected the Enter key to work for moving between the login fields.

Users expected that after they entered their login they could click Enter to move to password, and from password to the actual login. All five users commented on that.One user had another kind of problem: after entering her login she clicked Enter twice and was dropped into the error message: "Forgot your password?"One user wanted to use Tab or Enter to be able to move from field to field. So it would be advantageous that both Tab and Enter work for moving, but Enter is more crucial, because more users would like it to have this functionality.Recommendation: Pressing the Enter key should move the cursor from login to password and from password to enter (but not from login to enter).


Frame/No Frame/ My Defaults were unclear for users.

None of the users was able to confidently describe the function of the Frame/No Frame/My Defaults switches located below the login. Three of the testers guessed that the switch would determine the view of the application. However, after they chose one of the options, nothing happened, so the users assumed that just switching is not enough but did not attempt to figure out what else is needed. One user commented that "after working with the application for about 50 minutes already, I still do not understand what is required to make them work and don't care to know."Recommendation: Make the action of the switches immediate or remove them, because people are not using them as they are now.

Forgot your password? feature was appreciated by users.

All users noticed the feature and said that they liked that it was there.Recommendation: n/a

Visitors and Info sections

First-time users read the information sections very carefully.

Users always spent quite a bit of time reading. Users read Policy, Features, FAQ, and Advertising carefully. I had to stop each of the users after some 30 minutes of exploring visitor information to be able continue the test.Testers overall were quite suspicious to the information. Several of them asked me what is the catch.On the average users did not believe the claims that their personal information will not be disclosed to anyone. They seemed to be overall untrustful of the security of Internet information, especially the security of personal data. They also suspected that their personal correspondence will be accessible to some Hotmail personnel.The fact that the email was free made inexperienced users very suspicious; they mentioned that there must be a catch and did not quite believe my explanation that the money comes from the advertisements - "this should not be enough, what's the catch?" They also mentioned that if the money comes from advertisement, that means that more and more of them will be present in their email.The fact that the Hotmail tool belonged to Microsoft added to the suspicion. Overall it appears that attitude towards Microsoft is very negative in this area (Midwest), especially among UNIX users.Users said that Favorite Quotes and Press Release are not worth looking at, because they probably contain only "promotion." So if the Hotmail authors want people to read the information, they should consider renaming these sections.Anywhere the text seemed to be catchy and gimmicky, they became more unbelieving. Phrases like "no gimmicks" only made matters worse.Recommendation: Try to avoid catchy phrases. Do not make users' information available to anyone.


Users like the layout and information of the Features section.

Users liked that there is a picture and that the explanations are clear.Recommendation: Use graphical explanations where possible.

One user was wondering how to terminate her account.

Recommendation: Provide an explanation about the procedure.

The initial interest to Hotmail was high.

Users were interested to try the free email. They thought that it was very convenient that it is available everywhere.Recommendation: This is a good feature to use in promotions.

Information presented in the Visitor Information sections seemed adequate.

Users seemed to be satisfied with the amount and content of the information provided. Several users went to the FAQ. One user read the FAQ very carefully. It seemed users were satisfied with the amount of information it covered.After spending some 30 minutes reading and based on the information they read, users felt somewhat secure about their decision to continue using Hotmail.Recommendation: Amount and content seem adequate.

Registration

Hotmail Terms were read carefully. Disclaimer seemed too long.

Users read very carefully the Hotmail Terms (from the "Accept" page). Their most serious concern was the security of the information they provide when getting a login. After reading the text users still thought that their personal information will be sold by Hotmail to others for promotion. Additionally two users thought that the content of their correspondence would be accessible to Hotmail personnel.Users did not like how long sections 7 and 8 were - DISCLAIMER OF WARRANTIES and LIMITATION OF LIABILITY - "too much information we have to read about them," "they are just protecting themselves from everything."All in all, information presented in the Terms section seemed fair.Recommendation: n/a

Users did not like to enter personal data.

People did not like to give their age and location. They did not understand why occupation mattered, besides there was no correct occupation for four out of five testers.Recommendation: Is this information really necessary? I suggest making it optional.

Positioning of the Clear button in front of the Submit button confused users.

Two users almost clicked on Clear instead of Submit.


Recommendation: It would be more logical to put Submit registration first and Clear after that.

Error handling: why the password is dropped?

If a user does not enter all information, the form returns with an error message and requests more information, and drops off password. Three users had this problem. Two users did not even notice the password disappear and were surprised when they were told that they did not enter it.Recommendation: Keep the password unless there are security reasons that prevent this.

The application needs to inform the user about all mistakes at once.

Recommendation: If the application notices several problems in the form, it should list all of them in one message, not one at a time.

Login

Login confirmation surprised one user.

One user was surprised that they asked her to retype her login to confirm her login information. She thought that it was rather strange because she just chose her login in the previous page, "so it should know it;" and she was in the process of registering, "so they shouldn't doubt that I want to register. What's the problem?" The other users did not have such an apparent problem with it.Recommendation: Retyping the login may be useful to help users remember their login; I would leave this feature, it just seems a little odd. On the other hand, if there was an apparent way to find your login, the feature could be eliminated.

A login search would be helpful.

It is feasible that users would forget their logins, especially because the logins of their choice are usually unavailable because are taken by other users, so they have to go with a new login, which may be hard to remember.Recommendation: It would be good to have a login search right from the Login page. It could be combined with an email address search (certainly it will only be available to those who registered).

The login choice may need more hints.

None of the users was able to choose a good login at the first attempt. The login they chose was always already in use.Recommendation: Add more specific instruction about the login choice that would recommend how to choose a unique login.


Users appreciate that the tool suggests logins.

Users appreciated that the tool suggested several logins when their own choice failed. They assumed that the tool suggests logins that are not currently used.Recommendation: Keep this feature, suggest only unique logins.

Users interpreted the Password hint incorrectly.

It seemed that users had an expectation that the password hint would help them remember the password, not to login without it. For example, one user who used her husband's name and some numbers as the password, used the question: "Husband's name and day." She than was going to enter her password as the answer, but did not like that the tool displayed it, instead of replacing characters with asterisks, so she changed the answer to "********". Another user seemed to understand the term and entered "What color is my car?" as the question, but put her password as the answer. Only one out of five users understood how to use the hint, but she used a phrase as the password answer, and later could not remember what was the phrase exactly.Recommendation: Password Hint should be renamed into Hint. The instruction should not imply that the hint will help them remember the password, but should say that the question and answer will be used instead of the password. Another alternative - because users expected that the hint will help them remember the password, perhaps the password hint question should be used. It would remind the users what they used as a password, but would not ask for the answer. In response to the question, users will enter their password, it won't be shown, and if it's correct they will be able to go into their email. They will not need to clear their cache because their password was not displayed.

Navigation - the left bar

Inbox, Compose, and Addresses were easily found, probably because many mail tools use the same term.

Folders were found and used easily.

Users had no problem discovering Folders and understanding how to use them. The concept of sorting into Folders seemed somewhat new. Experienced users commented that they use folders to sort messages into them after they receive them. I suspect that the task I gave in the test got users thinking about Folders, and that they may not find this feature on their own.Recommendation: n/a. I think this is a very useful feature and users will catch on with time.

Options attracted a lot of attention.

All five users found the functions located in Options without a problem. Two users demonstrated a tendency to go to Options for all functions: dictionary, member directory, etc. One of them explained that in her mail tool all actions are given through Options.


Recommendation: Option is a descriptive name. The Options page is organized well. It may be logical to provide all functions, for example POP Mail and Related through the Options menu.

Logout was not a problem.

Logout was not a problem for at-home users. They usually close Internet and disconnect. The corporate users, who usually leave their Internet running even when they leave for the day, were uncertain if it was necessary to close the window and log out. The consensus was that when going to lunch they would leave it running, but lock the screen; when going home would either do the same, or close the related Internet window.Two users clicked the logout button - "to see what it does." They were not sure what page it dropped them into, but were sure that they closed their email.Recommendation: The Logout button is not needed because users can easily do without it. Dropping into the MSN page may not be a good idea, because many users have never been to it before, so it was a surprise for them that they ended up there; also the MSN page is very messy.

News, Classifieds, and Help Center

The test instructions did not require users to use News, Classifieds, etc. Users did not show interest in looking through them. It seems surprising to me that Help Center fit into the same category with them. Users did not use Help Center.Recommendation: Consider separating the Help Center.

InBox

POP mail located in the Inbox confused people.

When in the InBox, uses noticed POP mail, did not have a clue what it was, but thought that it was related to their Inbox. They opened it and spent quite a bit of time just to figure out that it is not so related.Recommendation: Make POP mail available from the Options page only. Consider using the complete name, not the abbreviation.

The content of Related is different on different pages; users do not understand why.

Three users noticed that Related is different on different pages. They commented that it is unclear to them why its content is different and how to get to the page where they can find Hotmail Members Directory instead of Thesaurus. So they did not understand the logic of organization of the Related area (although they understood that there was logic to it).It seemed that users did not pay attention to the word "Related." They said something on the order of: "I like that you have a Thesaurus there, in the right corner." I do not think that they did not notice the word; it seems that its meaning was not clear or relevant.


Recommendation: Keep the content of Related the same in all windows. The title "Related" may not be very explanatory. I understand your reasoning that Thesaurus is useful when a person is writing or reading an email, while Member list is needed in the Inbox. But this understanding may be incorrect and I need to spend time thinking about - it is not apparent. It is always good to put things in one location and keep them there at all times, or users will not be able to find them.

It appears that member directory is not available to people who are not listed in it.

A user who requested that her address is not listed in the member directory was not able to use it. She thought that she should have been warned about that when she made her choice to not register.Recommendation: Inform users about this implication. Additionally, it is not clear to me why these restrictions should apply. This person was still a Hotmail member, so why couldn't she use all services?

The difference between Dictionary and Thesaurus may not be obvious to all users.

One user asked me the difference between Dictionary and Thesaurus. Others did not have this problem. However I can see how this can be unclear, especially for nonnative speakers.Recommendation: It would be better to replace Thesaurus with the word "Synonyms." It also may be useful to replace Dictionary with "Spelling."

The function and purpose of the Microsoft Network switch was not clear to users.

Two users started looking at the pages available from the switch, got lost, and did not know how to come back (especially because the URL is cryptic). Another user checked it and assumed that it was a Bookmark feature, but did not see the point of using it, because he/she already had Bookmarks.Recommendation: Consider removing the pulldown altogether or rename it to accurately describe its function.

The mailbox organization is good.

Users liked the overall organization of the mailbox. None of them had a problem understanding it. One user specifically mentioned that she liked that Subject, From, and Date were clearly seen, unlike in her mail tool. One user commented that she liked the Delete button to be out in the open unlike in her own email tool, where the only Delete button available is the one on the keyboard.Recommendation: n/a.

Sorting was discovered by four our of five users.

It is hard to tell if sorting is something that people would use on their own. But when asked to organize messages by sender alphabetically, all five users started to look for a way to do it through clicking on the From column header.


Three users were able to sort without problems. One user was not able to sort by anything but date. She moved her mouse over the From header, but apparently only the part around the word From was "hot," the rest of the heading was not, so she thought that the whole cell was not clickable. Because the Date table cell is so small, pretty much all of it was a hot spot, so she though that she could only sort by date.Another user was able to discover the feature only after a significant delay. The problem was that he already tried clicking table headers in the address book. The table headings there - Nickname, Name etc, - are not clickable, while in the Inbox they are (New From, etc.). This inconsistency delayed him from discovering the sorting function.Recommendation: It seems that clicking on the table cell where the column head is located is a natural way for people to sort, so the overall organization of sorting seems to work fine. To facilitate discovery of the sorting feature for novice users, make the complete cell clickable, not just the word itself. Everywhere in the application avoid similar looking features that work differently. For example, if sorting is not possible in the address book, the table should not look similar to the one which can be sorted.

It may be preferable that the application keeps track of users' preferences in sorting.

It bugged one user very much that the default setting was to start with the earliest date. He tried sorting by date, but it took two clicks to sort the way he wanted - about 15 sec.Recommendation: Users would get less frustrated if the tool would remember their sorting preference and present the mailbox in the way they prefer it. (I understand that one of the reasons for this is that email storage space for each user is limited. However, it's still good to give the user a choice). Additionally it is more conventional to sort in an opposite way from one click than to click ascending and then descending.

Some users wondered why the size was shown. One user asked what those "1k" were.

Recommendation: It may be helpful to explain this in more details.

The use of the column after New was somewhat mysterious for users.

Users figured out eventually that this column is for selection. Users did not like that they could not select by just moving the mouse over the fields.Recommendation: The current selection is okay; provide mousing over selection if possible.

Users understood and liked the "Move to" feature and the concept of filters overall. See more discussion on filters later.

It is not clear why names were chosen to serve as links to the email messages.

It seemed that users had some confusion as to how to open emails. One user asked why names were chosen to serve as the link to the email. It is my speculation, but I think that this may be confusing. In other tools, for example Netscape mail, names (when clicked in the body of the message) lead to the address book, not to the message this person sent. It may seem bizzar that one may have five emails from the


same person each with a different subject, but it would be by clicking at the person's name that the users would actually get to the emails. The more so, because in the address book, clicking on the name actually starts an email to that person.Recommendation: This is not a serious usability problem. It probably does not need to be taken care of, but as an option, the Subject of the messages, not the From can lead to the message.

Message that people already read remained blue

Messages that people already read remained blue if they read them during their previous login; this seemed confusing to one person. (I cannot claim that it was not confusing to the rest of the testers, because they simply may not have said this out loud.)Recommendation: Consider not changing the color of links at all (at least not in the From field).

Compose

Compose does not have the left navigation frame.

One user was seriously upset that Compose was opened in a window that was missing the left navigation bar. He said that it drops him out of the application. He thought that it would be very convenient to see where he is and have all the controls available. He wanted Compose to open as a separate small window on top of the Hotmail window.Recommendation: This user had similar concerns about all windows that were missing the left navigation bar, and did not have any concerns about the windows that had it. This is why I think just adding the navigation bar to all windows would help.

Bcc is unknown to many people.

Two testers wondered what Bcc was. None of the testers used it.Recommendation: Explain it.

Users did not know how to send emails to multiple addresses.

It appeared to me that because there is just one line for To, while CC and Bcc have separate lines, it would not be apparent for users how to send emails to multiple addresses. In one of the test tasks I asked users to send an email to two people. Three users sent separate emails, one user sent a CC to the second addressee.Recommendation: It may help to provide several fields on the To line, so that it looks like more than one address can be used.

Sending and receiving attachments is usable.

All users were able to figure out how to send/receive attachments; three out of five users had problems, but were able to recover. One user, after she received an attachment and was asked to send the same file to her friend, just forwarded the email to him, the attachment was not sent. One user was wondering where the file she


received as an attachment was saved to. She did not know that she can click on the attachment to open or save it. It is inconvenient that attachments do not work in IE 3.0. Users liked that the Attachment button is located right on the Compose screen.Recommendation: Make functionality available on older browsers. Try to facilitate user problems; with more use, users will be able to figure out the correct way of sending attachments.

Organization of the Compose Window seemed fine.

Users liked the organization and layout of the Compose Window. One user said that she likes that the controls are located so apparently. Functions of the controls seemed clear to users. Only QuickList caused questions; users tried using it, and understood that it provides fast list of address, they liked that.

The format of the Dictionary and Thesaurus buttons is inconsistent.

One user commented on the inconsistency: in the Compose window, Dictionary and Thesaurus are buttons, while in other cases they are just links.Recommendation: Make consistent.

Address

Users did not understand the term Nicknames.

It was hard for users to understand that in the application this term has a special meaning. Especially because users first face it when they create their first entries in the address book and discover that Nickname is required. "What if this person does not have a nickname?" Later when the user sees the address table with Nicknames and names in it, and the Inbox with people's nicknames there, the meaning of nickname becomes clearer.Recommendation: Either provide an explanation right by the Nickname in the address entry form, or develop a different term. Consider also doing without it, for example use First or Last name instead.

Create, Edit, and Delete caused minor mistakes.

Users were able to find Create, Edit, and Delete, but the location of these features seemed to cause some mistakes. For example, when asked to make corrections in one of the entries, three out of five users clicked or almost clicked on the Nickname instead of Edit. The more so because in Folders, clicking on the folder actually allows you to view it, not edit. This inconsistency in functionality seemed to be the reason that users were unsure what to expect when clicking on different links.Recommendation: It may make sense to move the Edit and Delete columns closer to the Nickname column. Because it seemed natural for users to click on the Nickname to edit the entry, perhaps this feature should be assigned to that field. And the function of sending an email to the person can be assigned to the Send to button located similarly to how the Edit button is located now. There also may be a more elegant solution that


would avoid using the words Edit (or Send to) and Delete for every person, but somebody has to think about it.

Groups is a good term.

Groups is a good term, it was clear for all users, even those who use different words for group addresses - "alias," "list," etc.Recommendation: Keep it as is.

Users liked the confirmation message, but it can cause confusion.

Users liked the confirmation "The message has been sent" a lot. They also liked the opportunity to turn it off, but said that they wanted to keep it on. However, when users sent messages to an incorrect address, they still received the confirmation. Later they saw that their message did not go through. Three users did not know what happened.Recommendation: n/a

Compose can be located off InBox.

Two users were looking for Compose in the InBox, but then found it on the left bar.Recommendation: Compose can be located off Inbox, this way it won't take extra space on the left bar.

Addresses - create

Nickname

Nickname was confusing. See above.

Users had different opinions about the number of fields in the Create Address window.

Four out of five users did not have a problem with the number of fields in the Create Address window. They knew that they are not required and are for their use only. One user specifically commented that she liked these fields because she will have the information handy there. The fifth user did not like that the application asked him for all the information. He filled the fields with bogus data. He also was sure that the information that he entered will be available to Hotmail and will be sold by them to others. His problem could be in part caused by the fact that in the registration form this information was not optional.Recommendation: Make sure that users understand that the fields are optional.

Options

Personal, Password, and Signature

These functions were not tested.


POP mail causes confusion.

None of the five users knew what POP mail was. After reading the explanations, all users said that they did not know their POP servers and would not know how to find out. One user remembered that she read in the new user information that if a company has a firewall, connection to the company account is not possible. Another users after reading Help, concluded that using the POP mail feature costs money.Recommendation: If this is true, there should be a big note about that on the POP mail page. This would also make this feature less useful. If this is untrue, new user info needs to be changed.

Filters descriptions talks primarily about trashing junk mail.

I asked users to put messages coming from one person into a separate mailbox. Four out of five users were able to do that. However, the Filters description talks primarily about trashing junk mail, so one user was going to sort the message of this person into trash and read from there. While working with the Filters she discovered that she can set up additional filters and did that. One user asked if she will be notified if she has new messages in folders.Recommendation: The instructions should not talk only about trash, but suggest creating other folders. Consider providing links for creating folders directly form the instructions. Consider developing a notification mechanism.

Controls for setting up filter seemed awkward.

Users did not have serious problems using the Move, OK and Apply buttons; but their location and functions seemed odd to all five users.Recommendation: I believe there should be a better way to setup these controls. For example, consider combining OK and Apply buttons or rename them to clarify their functionality. Also place the OK button at both the top and bottom of the page. Consider a different way of prioritizing filters, for example by allowing users to number them, or allowing to WYSIWYG move them.

None of the users noticed the link to Folders on the Filters page; they used the filters link from the left bar.

Recommendation: The instructions should not talk only about trash, but suggest creating other folders. Consider providing links for creating folders directly form the instructions.

The concept of filters is new, but useful.

The concept of Filters seemed somewhat new, even to the experienced audience. I suspect that the task I gave in the test got users thinking about Filters, but they may not find this feature on their own. Users liked using filters.Recommendation: I think this is a very useful feature and will catch on with time.


Preferences

Preferences were not tested specifically, but three users checked them out. The information seemed clear.

Overall

Overall users had no serious problems using the tool. Users referred favorably about it.

Recommendation: Take care of the details to make it perfect.

The site does not provide means for alternative navigation.

Based on this and other tests, I know that users have different preferences in navigating web sites. Many people like to use URLs to go between pages. Then when deep in the site, they like to go to previous pages by just deleting part of the URL of the page they are at now. The Hotmail URLs do not provide this opportunity; the URLs are quite cryptic, several users commented on that.There also is no Home or "Back to login page" button. We needed it in the test, I am not sure if in normal situations people would need it, perhaps only to be able to read the new-user information. If the URL was meaningful, the need for the home button would be less.Recommendation: provide a somewhat meaningful URL for the pages and keep it the same from page to page and just add to it; for example, http://hotmail.com/Inbox. If it is impossible due to the way the site is programmed, then it may be important to make other means of navigation more apparent, for example provide a Home button.

Text links seem unprofessional.

One user was seriously upset that the application contained links, not buttons. He thought it looked unprofessional because of that.

Two hot spots are unnecessary.

One of the users claimed that she cannot click on the words Inbox, Compose in the left frame, she said that only the colorful bar next to these words was a hot spot. The same user thoughts that New, From, Subject, and Size in the Inbox were not clickable.Recommendation: it is not clear why two hot spots are needed; replace with one.

Advertisements in their current state do not bother people.

Users were not annoyed by the advertisement, so we can assume that the amount of advertising is not overwhelming. However, users were concerned that the amount may grow.Recommendation: it is desirable that the amount of advertising not increase.


It is hard to find previous ads.

One user got interested by an advertisement that the site had. She checked it out, then returned to the test tasks. After the user logged out and logged back in, she wanted to go back to the ad, but was not able to find it.Recommendation: It may be useful to provide a place where users can find old ads.

Moving ads are annoying.

One users was seriously annoyed by a moving add; she tried to cover it.Recommendation: Provide a means of closing advertisements, because moving pictures distract attention. Or make sure that the rate of movements is fast only at first and then gets much slower (about once a minute).

Users did not like that the tool advertises Hotmail in their messages.

Two users did not like it, because they perceived it was an intrusion in their private mail.Recommendation: Avoid advertising in personal messages.

Help was used rarely.

Only one person used Help during the test. Help, when opened and iconisized does not reopen.

Frames work fine.

Users liked the layout of the page with frames, and preferred it over the non-frame version (there was no noticeable difference in speed). The testers found the way to change frame preference.Recommendation: Leave the frame version as a default.

Greetings distract attention.

Three out of five users clicked on Greetings and started wandering out of the mail tool, thinking that it is related to it. One user specifically commented that he thought that it related to his "greeting information" - address, or signature file.Recommendation: Greetings distract novice users form using email. It is not clear why Greetings is considered so important that it has its own button, but maybe there is a reason. Consider moving it to the left frame, out of the email pages. Other possibilities are reducing the size of the link, or explaining Greetings as an alternative way of correspondence.

Greetings have some usability problems.

Greetings do not recall names from the user's address book.Back to Hotmail from the Greetings page didn't work.


Use of color is not meaningful, may even be confusing.

The site does not take advantage of colorcoding: colors do not convey any meaning or relate similar objects. There is a multicolored bar on the right side of the left (controls) frame; it seems that is it used only for attracting attention to the controls. It is not clear why these particular colors were chosen to represent different functions (no apparent connection between blue and Inbox, purple and Compose). But perhaps the tool could take advantage of this color coding by providing a similar color schema inside the Inbox and Compose windows. This would also indicate for the users where they are. Just writing Inbox at the top of the page did not prove helpful- two users clicked on the page headings "to see where these links lead" - so, apparently, they did not think that these were the page headings.Recommendation: Consider a more meaningful color scheme.

Users perceived any item that is blue as a potential link.

Users tended to click on everything that was blue or purple, or white on blue background, and were disappointed if it was not a link. This probably is happening because many web sites now break the standard that only blue underlined is a link. I observe this tendency in many usability tests, so I would recommend avoiding shades of blue and purple for anything that's not a link.Recommendation: It may be a little rigid, but try stick to the rule that all links are blue and all blues are links.

Stop was inactive during most actions.

Each time a user clicked on something, Stop became inactive, this was very inconvenient in several cases when the user suddenly changed his/her mind and wanted to stop the action. The slow speed of sites loading added to this frustration.Recommendation: If possible, make Stop active during actions and moves between pages.

Performance

Speed is insufficient.

The tool was unstable and crashed at least once during each test, even caused a Netscape core dump. The crashes occurred in different places and after regular user actions. A tool often popped up the message: cannot connect to the server try again later. Sometimes the user would be dropped out a page that would say "error." The tool was slow even on our corporate network. The tool sometimes hung - could not connect to the site, so just sat there with the cursor looking as a clock. Users became increasingly concerned about the tool performance and reliability as the number of such crashes increased.Recommendation: Try to avoid performance failures in order to keep customers.


It appeared that the tool behaved differently with different systems on different days.

On one machine, double-clicking the Enter key after entering the password led to the popup "Forgot your Login?" - so the enter key worked for her. On another machine Enter did not work to move between the login fields.Recommendation: It is recommended to avoid inconsistency, so that there is no surprise in the tool's performance.

Developers' questions

Note: The number of users and the ways users are chosen do not provide a representative user sample. Usability testing does not serve as a means for statistically valid approximations. Thus answers to these questions show just a trend, they may be correct for only one group of users or a specific geographical location. To obtain statistically valid results a bigger sample is needed.

How do users currently do email?

See Appendix 3.

How much effort do users have to expend to use Hotmail effectively? How satisfied are users with Hotmail?

It seems that during the first hours of work with Hotmail, users are quite satisfied with it. They generally find it more user-friendly than many of the systems they worked with before. However, it is possible that later, when using the system on a regular basis, users may dislike some of its features (for example, default sorting of messages by the date in ascending, not descending order). I do not have enough evidence to confirm that or predict what features can cause frustration.

If a first time user, do users think they will continue using Hotmail on their own? Would users recommend Hotmail to others?

At the end of the test four out of five users said that they liked Hotmail and would like to try it again in the future.

How reliable do users think Hotmail is? Would they want to use it for urgent or time-sensitive information?

Four out of five users said that they would use Hotmail for urgent or time-sensitive information. One commented that she would try it first to see how it works, another user commented that she would use it assuming she did not have problems with it previously. Note that if Hotmail continues to crash as it did during our test, this may be a reason for considering it unreliable.


Appendix 1: Test ScenarioCUE-2 - Scenario Version 8. October 1998

Scenario:You've been hired to evaluate Hotmail.com, the world's free email service you can access from any internet connected personal computer. The marketing management at Hotmail has sent you the following information which identifies their goals and objectives for this evaluation.

Part One:Hotmail was recently acquired by Microsoft and can now be accessed from both the following URL's: www.home.microsoft.com and www.hotmail.com.Management is very interested in knowing which avenue users are taking to reach their site, and whether or not users' perceptions of Hotmail (and MSN) are influenced by the avenue they choose.[Note: The original Hotmail.com home page and the logo in the top left corner of the navigation bar within the site will most likely change branding in early October. You are required to take a screen shot of the state of the hotmail.com home page you evaluate for the CUE comparison so we can know if different teams evaluated different branding designs.]If users are already familiar with Hotmail, how do they choose to access it? Is it easier for new users to locate Hotmail from one URL over the other? Is there any distinction in the kinds of users that tend to use the various avenues?Additionally, Hotmail defaults you to the frames version of their site (if your browser is capable of interpreting frames). Management is very interested in knowing whether users prefer the non-frames version.Hotmail's biggest competitors are: Yahoo Mail and Netscape WebMail.Do NOT test the localized versions of HotMail, like www.Hotmail.dk

Part Two:Management is also very interested in the general usability of their product and has identified some specific areas of concern. Below is a list of features that Marketing and Engineering have identified as benefiting from user feedback: (Listed in order of importance to Product Marketing)1. Registration2. Login3. Logout (what do users expect after hitting the logout button?)4. Viewing hotmail with or without frames - do users have a preference?5. Customization (under the "Options" section, particularly "Preferences" for viewing style)6. Performance issues (speed, etc.)7. Compose8. Password Retrieval9. Sending and receiving attachments10. Reminders11. Hotmail member directory and email lookup


12. Sorting messages in your InBox (by Subject, Date, etc.)13. Reading a message (related issues: Screen resolution and size; URL recognition when a message has a url in it))14. Address book15. Folders16. POP (under Options)17. Advertisements (general attitude, do users click on them? Why, why not?)18. When do users try to access Help (do not test help system, just identify when users feel they need help)

General questions which management is interested in addressing are:1. How do users currently do email?2. Generally, how much effort do users have to expend to interact with and use hotmail effectively?3. How satisfied are users with Hotmail?4. If a first time user, do users think they will continue using Hotmail on their own?5. Would user recommend Hotmail to others?6. How reliable do users think Hotmail is? Would they want to use it for urgent or time sensitive information?

HotMail management wants to focus only on the mail features and functions, and not the other services now offered from the site. They are *not* interested in the following areas:ClassifiedsWeb CourierNews and Linksor other general portal features.

User Profile:HotMail is a world-wide free email service targeted towards anyone who would like the convenience of managing their email on-line through a web browser. HotMail can be accessed from any computer or kiosk that provides internet access, making it easy to access your mail from anywhere. Hotmail also allows you to manage all your own personal email without it being recorded on your employer's servers.

Questions?If you have questions about product goals, user profiles, or product features, please send them to [email protected]. Erika will forward them on to the HotMail contacts, as well as distribute responses from them. Erika's role as proxy is to limit the number of duplicate questions being sent to HotMail management.

Test Scenario clarification

Subject: discussion?To: Rolf Molich <[email protected]>Rolf,


A part of my testing process is a discussion with the developers when they show me their product and explain how they want users to work with it - step by step. I noticed that this exercise is very helpful for me. Would it be possible to organize such a discussion?Half-an-hour on the phone would be good for me. Or perhaps they could write a step-by-step scenario?Thanks,(name deleted)

Subject: CUE-2, a couple of questionsTo: [email protected] Erika,I have a couple of questions to you or the Hotmail contact.1. I don't quite understand what the following phrase is the Scenario mean: "Management is very interested in knowing which avenue users are taking to reach their site."Does the management wants to know which URL do users discover first? Or does this mean that the management wants to know which URL would the users use if they were given both of them?2. Is there any marketing data on what group of consumers is considered the target audience. It seems to me that a person would not open an additional email account in the corporate environment. It is more likely that people would use Hotmail for personal use, from home. Thus it is likely that the mail's customers are not very computer literate and that their internet connections are slow. Should I assume that and find test participants respectively? Is there any information of that, or I have to take an educated guess?Additionally, what is the primary motivation of people using Hotmail: is it for the correspondence that they cannot do through work email, for reading news, for chatting, etc.? This information can be found by the marketing department, I don't think that I will do this kind of research myself.3. Please define the word "reliable" in Q 6 of general questions: "How reliable do users think Hotmail is?" Here reliable means: does not go down, easy and fast to access, error-proof? What do you really want to know?Thanks for your help,(name deleted)

Subject: Re: CUE-2, a couple of questionsFrom: Erika Kindlund <[email protected](name deleted),I've answered your questions in the text below. Please let me know if you need any further clarification!!!Good luck with your study :)Erika


I have a couple of questions to you or the Hotmail contact.

1. I don't quite understand what the following phrase is the Scenario mean:"Management is very interested in knowing which avenue users are taking to reach their site."Does the management wants to know which URL do users discover first?Or does this mean that the management wants to know which URL would the users use if they were given both of them?

I think this means, left to themselves, how do users navigate to hotmail...viathe microsoft site, or via a direct hotmail url. Don't give them either url....just see which one they end up using to access hotmail. Then see if there is a notable difference between the types of users who access Hotmail thru the microsoft site vs the hotmail url.

2. Is there any marketing data on what group of consumers is considered the target audience. It seems to me that a person would not open an additional email account in the corporate environment. It is more likely that people would use Hotmail for personal use, from home. Thus it is likely that the mail's customers are not very computer literate and that their internet connections are slow. Should I assume that and find test participants respectively? Is there any information of that, or I have to take an educated guess?Additionally, what is the primary motivation of people using Hotmail: is it for the correspondence that they cannot do through work email for reading news, for chatting, etc.? This information can be found by the marketing department, I don't think that I will do this kind of research myself.

Actually, corporate users are in fact one of the largest market segments. Most corporate employees are prohibited from using their corporate email for personal use (if you can believe that! we're lucky cause we're in the computer industry :) Other corporate users just don't want their personal or sensitive email living on their company's mail servers (actually, that's pretty smart business practice if you think about it). But also home users is a large segment. Hotmail is easy to use and doesn't require a novice user from having to configure their machine for email - it's all just done through the browser.It's safe to assume that corporate users are most likely using hotmail for personal use, and home users for all their emailing. Please feel free to recruit all user types - advanced and novice - and get their feedback on Hotmail. Maybe asking them about when they would and wouldn't use hotmail is a good question to include in the study.

3. Please define the word "reliable" in Q 6 of general questions: " How reliable do users think Hotmail is?" Here reliable means: does not go down, easy and fast to access, error-proof ? What do you really want to know?

By reliable we mean, would users use hotmail to send and receive very important information, or would they feel that their information might not make it in time or would get lost or would not be available when they needed to refer to it, etc...


Subject: Re: CUE-2, a couple of questionsFrom: Erika Kindlund <[email protected]>

I don't understand. What kind of task do you want me to give them: "You've heard that there's a mail tool called Hotmail. Please find it." ?

Currently there are 2 avenues for getting to hotmail. Going directly to www.hotmail.com or going to www.msn.com and typing in your hotmail login id in the email field on the msn home page.Marketing wants to know if there is a generalizable difference between the users who access their Hotmail account from the msn home page login field, vs. going directly to the hotmail page and the login screen there.Marketing has asked you (being the usability expert) to evaluate the two avenues of hotmail access to identify the types of users who gravitate towards one way over the other. Please use any tactic you feel is appropriate for evaluating the accessibility of the two avenues.Does this help???Erika


Appendix 2: Test Method

Method descriptionFormal usability tests were conducted with five test participants. Each of the tests was followed by an interview.

Method constraints

A note to the developers

Email is a difficult tool to evaluate with just one kind of usability testing. A formal usability test combined with an interview seemed most appropriate in this situation. However it had some deficiencies.

A formal usability test is best suited to describing the pattern of work of the first-time user. In all of my tests, it took at least 40 minutes to get to the Inbox. Many of the questions posed by the developers related to the more advanced stages of working with the tool - filtering, setting up POP mail, etc. It is apparent that to get to these stages, the user has to spend a considerable amount of time with the application - a situation which is not possible to achieve during one usability session. (I prefer not to conduct tests that take longer than 1.5 hours.) What would be more advantageous would be to give users the tool to use for several days - up to one or two weeks, and ask them to keep dairies, or to answer questionnaires on a regular basis.Because of the length of the test and the difference in the style and speed of work, many situations appeared with just one or two people, often due to the fact that others did not get to that point or chose a different route and did not come upon the feature.

With some features, for example filters, the interesting question is not only if the the tool is easy to use, but also if the concept of the function is clear. Do users even see the need in filters? Do they realize that this feature is available? I discovered that the feature is easy to use, but by asking people to use it I already biased them by attracting their attention to it.

When reading the test scenario, I could easily see where many of the questions would much better be answered by a survey and/or focus groups. Consider for example, the question about the user preference in accessing Hotmail. I do not see a way that a formal test would answer this question. But consider a survey that would ask people to report how they first accessed Hotmail, where they heard about it, and how they prefer to access it now. People would be referring to their past experience, thus the data will be more accurate than if they say how they would prefer to access Hotmail, also the sample would be bigger. The developers could then compare the answers to other characteristics of respondents and possibly would find the answer.


This is why I would like to suggest that the development team do additional testing after reviewing these findings.

A note to the organizers of the comparative study

It was hard to develop a test scenario that would cover all important aspects of this application. An email is a tool for everyday use, so it was important to evaluate both the first-time and advanced features. For the purposes of a comparison study, it may have been more advantageous to choose a shorter application with a more concrete set of tasks and fewer test questions.

Additionally, we were working in an artificial situation where a usability specialist was unable to talk to the developers. If I had the opportunity to communicate with the developers, I would be able to clarify the purposes, methods, and test questions. So we would either reduce the number of questions or use several different methods.


Appendix 3: TestersS1 - female, illustrator, uses Netscape Mail on a regular basis.

S2 - female, video engineer, uses Netscape Mail on a regular basis.

S3 - female, music teacher, uses some sort of email on a regular basis (she could not recall the brand of her email).

S4 - male, business consultant, uses Eudora and Lotus notes.

S5 - female, psychologist, uses Netscape Mail on a regular basis, previously used Microvolt mail tool (not Outlook) and Eudora


Appendix 4: Test booklet

InstructionsThe following pages contain tasks that we are asking you to complete using the web site that will be presented. We are very interested to see if the proposed web site design performs well and satisfies users' needs.

In order to understand what you are doing and thinking it is very important for us that you think aloud during entire test. Please feel free to express your opinion, give suggestions, discuss what you are observing. Describe your actions, your expectations and perceptions. Behave as if you were all by yourself and would talk to yourself or as if you were telling a friend about what you are observing.

Read the instructions for each task aloud. Go through this booklet at your normal speed. When you are done with one task go on to the next one.

Remember to THINK ALOUD while you are working.

Introduction1. You have heard about a new, exciting, convenient free mail tool. It's available through the Internet and you can use it to send and receive email and to receive news. This mail tool can be accessed from any computer or kiosk that provides internet access, making it easy to access your mail from anywhere, even from a different country or different city. The mail tool also allows you to manage all your own personal email without it being recorded on your employer's servers.Would such tool interest you? Why?

2. Have you heard about Hotmail? Have you considered using it? Why?

Test Tasks

Background

You have several friends with emails who you exchange emails with regularly. These are personal messages and you don't want them to end up in your corporate account.

Task 1

You've heard about Hotmail and want to try it. How would you go about finding it?


Task 2

Please try to find Hotmail using the information you know about it (see Introduction).

Task 3

When you find the Hotmail page, please familiarize yourself with the information that it provides as much as you would do in real life. Describe out loud what you are doing. When you feel that you have seen enough, please go on to the next task.

Task 4

What would you do at this point:1. Quit the web site because you need to think whether you want to use it or not and maybe return to it later?2. Start setting up the mail tool for yourself?3. Quite the web site and decide not to use Hotmail.4. Other?

Task 5

Please explain your choice.

Task 6

You liked Hotmail and decided to use it.Please go through the setup procedure (if there is any) and send email to your friends (name deleted) and Andrei.(name deleted)'s email is XXXAndrei's email is andrei-k@YYY

Task 7

Explore the mail tool as much as you would do in real life. Set it up as you would like it.

Task 8

Log (name deleted)'s and Andrei's emails into the address book.

Task 9

Time to go to lunch!You will return to your email later.


Please come to the observation room. Before you stand up, please don't forget to take off the microphone.Leave the email tool as you would leave it before going to lunch.

Task 10

Several hours have passed before you were able to get back to your email.Check if you have any mail. Please open some of the messages and comment on them.

Task 11

Oh, it turns out that you've remembered Andre's email incorrectly. In fact it is andrei_k@YYY, not andrei-k...This means his address is wrong in your address book. Please fix that.

Task 12

Check out what (name deleted) sent you.

Task 13

Oh, this is nice file. I bet Andrei would be glad to receive it. Please send it to him.

Task 14

Did you know that yours and (name deleted)'s good friend Joe Johnson also has an email with Hotmail? Send an email to him.

Task 15

Time to take a break from mailing. You'll get back to it tomorrow.This time you want to close the tool. Please do so.

Task 16

You are back and want to check if Joe replied to your message. But bummer, you forgot your password. You still may be able to check your email. Please do.

Task 17

Suppose you forgot your login. Can you find it?


Task 18

Check your messages.

Task 19

What is the last message you got?

Task 20

Please group all messages from (name deleted) together.

Task 21

(name deleted) sends you too many stupid messages. It would be good if you could send them to a separate place in your mail box and retrieve them from there when you feel like reading them.Can you arrange for them to be received in a separate mail box?

Task 22

What does Reminders function do?

Task 23

Please set up your POP mail.

Task 24

You are curious how this site would look without frames. Please switch to a non-frame version.

Task 25

What does the My default switch do?

Task 26

To conclude, we would like to ask you several questions about your experience today. (name deleted) will be there in your room in a second to ask them.


Appendix 5: Interview questions1. How do you currently do email?

2. Generally, how much effort do users have to expend to interact with and use Hotmail effectively?

3. How satisfied are users with Hotmail? Will you use it after this test?

4. Would you recommend Hotmail to others?

5. How reliable do you thinks this email is?By reliable we mean, would you use Hotmail to send and receive very important information, or would they feel that their information might not make it in time or would get lost or would not be available when they needed to refer to it.If you have other email accounts, would you prefer Hotmail over them for such important information?If you had own business world you use hot mail as the emailing tool? (In the beginning no, later - yes)Would you use it as your children's email?Would you use it for urgent or time sensitive information?

6. Have you heard of Hotmail, Yahoo Mail, and Netscape WebMail.

7. How did you like advertisements?

8. Hotmail defaults to the frames version of their site (if your browser is capable of interpreting frames). Do you prefer the non-frames version?

9. If users are already familiar with Hotmail, how do they choose to access it? Is it easier for new users to locate Hotmail from one URL over the other? Is there any distinction in the kinds of users that tend to use the various avenues?Using through Microsoft: http://home.microsoft.com


Appendix 6: Screen shots


usability report - dialogdesign · web viewformal usability testing was chosen as the method of...

Documents