onetravel final report onetravel

!!

!

!

!

! !

School&of&Informatics,&IUPUI&

OneTravel&Team&I543&Usability&and&Evaluation&Method&&&&Final&Report&

12/10/2012!!

Instructor:**Michael&Wilson*Team*Members:*Aeshvarya&Verma&Powit&Rungsangthiwakorn&HsiaoNWen&Wu&Reecha&Bharali&

I!!

Content*List of Figures ................................................................................................................ I

List of Tables ............................................................................................................... III

1. Executive Summary ............................................................................................... 1

2. Introduction to the Testing ..................................................................................... 2

2.1. User Demographics ................................................................................................... 2 2.2. Testing Procedure ..................................................................................................... 2 2.3. Task Conditions ........................................................................................................ 3

3. Overall Results ....................................................................................................... 5

3.1. Success rates of across all tasks ................................................................................ 5 3.2. Time on tasks for all tasks ......................................................................................... 7 3.3. Post-task ratings across all tasks ............................................................................... 8 3.4. System Usability Scores (SUS) ................................................................................. 9

4. Usability Testing Results ..................................................................................... 11

4.1. Homepage ............................................................................................................... 11 4.2. Loading page ........................................................................................................... 13 4.3. Search result page ................................................................................................... 14

5. Synopsis ............................................................................................................... 21

5.1. Usability Problems .................................................................................................. 21 5.2. Recommendation .................................................................................................... 22

6. Appendix .............................................................................................................. 24

6.1. User Testing Package .............................................................................................. 24 6.2. Quantitative Data .................................................................................................... 31

!

!

!

!

!

!

!

II!!

List*of*Figures*

Figure 1. Accumulated success rates across tasks. ........................................................ 6!

Figure 2. Checkout button in OneTravel wesite. ........................................................... 6!

Figure 3. Checkout path in Hipmunk website. .............................................................. 6!

Figure 4. Average time on tasks among OneTravel and Hipmunk. .............................. 7!

Figure 5. Overall perceived difficulty across tasks among OneTravel and Hipmunk .. 8!

Figure 6. Overall post-task results for OneTravel. ........................................................ 9!

Figure 7. SUS score of each user for OneTravel and Hipmunk .................................. 10!

!

III!!

List*of*Tables*

Table 1.Success rates across all tasks and users in OneTravel and Hipmunk sites. . 5!

Table 2.Overall score of post-task questionnaires in OneTravel. ............................. 9!

Table 3.SUS scores of each participant. ................................................................. 10!

!

1!!

1. Executive Summary

OneTravel is a travel reservation website. The primary purpose of the website is to provide a single point reservation service to different travel booking, for example flight booking, hotel booking and car rental. The customers can reserve these services individually (Flight Booking, Hotel Booking or Car Booking) and they can also book all 3 of these services together (Vacation Booking). The latter is the main area that is being evaluated.

Expert Inspections were conducted on the website to preliminary identify the issues. Many of which was confirmed with real users testing the website. In addition, Hipmunk were tested side-by-side to illustrate the performance of OneTravel against the competing website. In the test, users are given basic task such as Finding Flight with a specific time, Finding Hotel with a specific location. Success rate, time taken to complete and other data, were collected.

The result shown that OneTravel underperformed when compare to Hipmunk in term of success rate, time-on-task and perceived difficulties. Results from observation have confirmed that serious usability, uncovered during inspection, exist within the website. The findings show serious problems in 3 main basic functions, Finding information in the website, Comparing hotel and Comparing flight.

Finding information in the website: 9 out of 10 users never used or notice the Vacation booking option. This clearly represents two important issues due to these two reasons. There are too many competing pieces of information and “Vacation” is not an adequate label.

Comparing Hotels with location constraint: The current structure of the website make it difficult to identify the location of the hotel, let alone comparing them. Users had to check them one by one and use additional tools, such as Google Maps, to help them complete the task.

Comparing Flight: All users have difficulty when choosing and comparing flight. OneTravel providing unrequested information, such as Near-by airport, prevents user from finding the desire information. In addition, booking flight that depart in a particular time, has a serious issue because of the structure and mechanism of the website.

!

2!!

2. Introduction to the Testing

Expert Inspections and user testing were conducted on OneTravel.com. The main focus of this test is in Vacation booking. Careful and extensive analysis through usability testing has revealed problems with the websites navigation, layout, information architecture and visuals. The user testing was conducted to better understand the heuristics. This is both to verify the findings and possibly uncover new issues that the users actually face while using the website.

Since OneTravel.com is a commercial website, Competitive Analysis was conducted to illustrate OneTravel performance against another competing website. At the time of analysis, Hipmunk is one of the best travel reservation website in the market and therefor selected for this analysis. With an emphasis on finding the usability issues associated with Onetravel.com tasks were designed to cover important aspects of the website through the results from the Expert Evaluation. These tasks are based on crucial functions such as finding information in the website, finding and comparing flight and finding and comparing hotel. Both the websites were critically analyzed and tested to make sure that all the tasks could be completely completed successfully within a respectable timeframe. Detailed test instruments and plan can be found in appendix section 6.1.

2.1. User Demographics

10 participants with following demographics were recruited. • 5 female and 5 male participants • English Fluency ranged from Fair to Fluent. • 6 hours and higher Internet usage. • Online Experience: 6 Competent and 4 Beginner 4 users. • No Experience with neither OneTravel.com nor Hipmunk.com • All have used a travel reservation website

2.2. Testing Procedure

The study was conducted with one facilitator and one observer during the active user session. 5 tasks, in 2 scenarios, were given to the user to complete. In order to avoid order effect in the result, 5 users start the experiment with OneTravel.com and another 5 users start with Hipmunk. In this experiment both qualitative and quantitative data were collected. Quantitative data consist of, success rate, time-on-task, perceived difficulties and System Usability Score (SUS). Qualitative data consist of observation, think-a-loud protocol, post task interview and post-test interview. In the testing session, these steps are being followed:

1. User is given a pre-test questionnaire. 2. Facilitator begins the recording and start the introduction script

!

3!!

3. Facilitator begins asking warm up questions 4. Facilitator gives user the Task Sheet. 5. Facilitator read the Task to the User 6. User begins Task 1.1 for website A 7. After user finished the task, he/she is give a post-task questionnaire 8. Facilitator begin a short-interview about the task completed 9. Repeat step 5 to 8 for all tasks 10. After completing all the task for website A, user is give a SUS

questionnaire 11. Repeat step 5 -10 for website B 12. Facilitator conduct Post-Test Interview 13. Stop the Recording and thank the participant.

2.3. Task Conditions

Tasks were designed to foster different needs from 2 different scenarios, cost conscious customers and high preference users. There are many severe usability issue within these discover within these area of function.

Scenario 1:

First scenario represents a cost conscious type of trip searching. In this scenario, user wants to find the cheapest trip possible to New York. To achieve this user will have to find cheapest flight and hotel and try to compare them in multiple days. This is to test the workflow of the process as well as the main function of the website

Task 1.1 Find the cheapest Flight and Hotel Task 1.2 Continue to make payment to the trip Task 1.3 Compare trip in different date

Scenario 2: The second scenario is designed based on the basic needs of the users, the ability to select a convenient departure time of a flight and the ability to select hotels based on the location. Task 2.1 Find Flight that depart at a specific time Task 2.2 Find Hotel that locate in a specific location

!

4!!

2.4. Severity Rating

The severity was calculated on a 4-point scale, 4 being a highly severe issues and 1 being a minor issue. The severity was determined by the error rates, possible recovery, recovery time and, most importantly, impact to the users.

1 = Cosmetic problem only: little or no impact to the user 2 = Minor usability problem: fixing this should be given low priority 3 = Major usability problem: important to fix, so should be given high priority 4 = Usability catastrophe: imperative to fix this before product can be released

!

5!!

3. Overall Results

3.1. Success rates of across all tasks

For all task completed by the participant, we defined 1 for success and 0 for failure. Table 1 shows success rates across all tasks among users in two websites. Figure 1 depicts accumulated success among tasks for two website. From these results, we found that Hipmunk has higher success rate in task 1, 3 4 and 5. These data reveals that users fail complete most critical tasks in OneTravel. Figure 1 shows that task 1 of OneTravel has the lowest success rate. This is because when finding flight and hotel together, users have to use ‘vacation’ button. However, most of users do not know the meaning of this button. They regard this label indicated travel related with traveling and relax.

OneTravel website only get higher rate in task 2 which is to find a way to

checkout purchases. The phenomenon was due to the design layout for payment process. Since OneTravel uses conventional pattern such as a button for checkout and a form for filling out, users are easily to recognize the section that they could perform this task. On the other hand, Hipmunk uses a dialogue box and direct users outside of Hipmunk. Figure 2 shows checkout path in OneTravel and figure 3 shows checkout path in Hipmunk website.

Table 1.Success rates across all tasks and users in OneTravel and Hipmunk sites.

(O for OneTravel, H for Hipmunk)

Task1 Task2 Task3 Task4 Task5

User Find Flight and

Hotel Find the how to

pay Compare Price

Find Flight w/ specific time

Find Hotel w/ specific location

O H O H O H O H O H

P1 0 1 1 1 1 0 0 1 0 1 P2 0 1 1 1 0 1 1 1 1 1 P3 0 1 1 1 0 1 0 1 0 1 P4 0 1 1 0 1 1 1 1 1 1

P5 0 1 1 1 1 1 1 1 1 1 P6 1 1 1 1 1 1 1 1 1 1 P7 1 1 1 1 1 1 1 1 1 1

P8 1 1 1 1 1 1 1 1 1 1 P9 1 1 1 1 1 1 1 1 1 1

P10 1 1 1 1 0 1 1 1 1 1

Success 5 10 10 9 7 9 8 10 8 10

!

6!!

!Figure 1. Graph showing accumulated success rates across all tasks.

Figure 2. Checkout button in OneTravel website.

Figure 3. Checkout path in Hipmunk website.

0!

2!

4!

6!

8!

10!

12!

Find!Flight!and!Hotel!

Find!the!how!to!pay!

Compare!Price!

Find!Flight!w/!specific!

Bme!

Find!Hotel!w/!specific!locaBon!

Num

ber!o

f!success�

Task�

Success!Rate!for!Each!Task�

OneTravel!

Hipmunk!

!

7!!

3.2. Time on tasks for all tasks

In each task, we measure time on task of all users. In order to have correct value, we subtracted loading time form measurements and then calculated average time on tasks. Figure 4 shows comparisons of time-on-task between OneTravel and Hipmunk. From overall results, we find that users spend more time on 4 tasks in OneTravel while compare to Hipmunk. But OneTravel had better performance in task 2. Users consume less time in OneTravel compares to Hipmunk when finding a way to pay. Because OneTravel provides a button at right hand side section, it is a short path for users to pay rather than go to the bottom of page and click next step. However, in Hipmunk, users had to find a way to pay in more effort because the interface is different to conventional way. Thus, they spent more time to figure it out in Hipmunk.

Comparing task 3 with task 1, we find that users can learn from tasks. In these

two tasks, they performed the search process both on flights and hotels. Moreover, they have to compare prices among two different departure dates. Ideally, it should spend more time on task when users have to perform same process for two times. With learning effect, they can be familiar with the interface and consume less time on second trial. From our results, the average time on task 1 of OneTravel is 8.5 minutes while task 4 is 6.0 minutes. It shows that users spend less time for performing search and compare. It is an evidence that they can learn from tasks.

Figure 4. Average time-on tasks between OneTravel and Hipmunk.



Compare!Prices!


Bme!


Onetravel! 8.5! 0.9! 6.0! 5.7! 5.0!

Hipmunk! 6.1! 1.5! 5.6! 3.1! 2.9!

0.0!2.0!4.0!6.0!8.0!

10.0!12.0!14.0!

Time!(m

in)�

Onetravel!vs.!Hipmunk!Average!Time!on!Task�

!

8!!

3.3. Post-task ratings across all tasks

After each task, we not only conducted interviews but also gave questionnaires to users. Users choose an answer from 1 to 7 in Likert scale. This questionnaire included three questions as following.

Q1: How was the overall task? (1 very easy - 7 Very Difficult) Q2: I was satisfied with the ease of completing this task. (1 Strongly Disagree - 7 Strongly Agree) Q3: I was satisfied with the amount of time it took to complete the task (1 Strongly Disagree - 7 Strongly Agree) In question 1, it asked how user felt about the task they just performed. This

question could reveal their perceived difficulty in the task. We compared OneTravel results to Hipmunk. Results are shown in figure 5. We find that there is not much difference between two websites in task 1. Average scores are 4.7 in OneTravel and 4.6 in Hipmunk which represent neutral difficulty. However, in task 2, since the interface Hipmunk is quite novel for users, their perceived difficulty is much higher than Hipmunk. From task 3 through task 5, perceived difficulty in OneTravel all are higher than Hipmunk. From learning effect perspective, comparing task 3, 4 and 5 to task 1, users felt not difficult as beginning. That is to say, they feel familiar with the interface as they performed more tasks.

Figure 5. Overall perceived difficulty across tasks among OneTravel and Hipmunk

By comparing results of three questions together, we find that the

relationship between satisfaction and perceived difficulty. From task ‘Find Flight and Hotel’ and task ‘Find the how to pay’, they are satisfied when they feel the task is easy. But they are not satisfied when they feel the task is more

0!1!2!3!4!5!6!7!



Compare!Prices!


Bme!


Average!Score!�

Task�

Perceived!Difficulty!

OneTravel!

Hipmunk!

!

9!!

difficult. Comparing to first task, task 3, 4 and 5 are not as difficult as task 1. Therefore, their satisfactions are higher than first task. Table 2 shows overall scores of post-task in OneTravel.

Figure 6. Overall post-task results for OneTravel.

Table 2.Overall score of post-task questionnaires in OneTravel.

Find Flight and Hotel

Find the how to pay

Compare Price



Q1 Q2 Q3 Q1 Q2 Q3 Q1 Q2 Q3 Q1 Q2 Q3 Q1 Q2 Q3 Average 4.7 3.1 3.2 2 6 5.9 3.9 4.4 4.5 3.2 4.3 4.4 3.5 4.7 4.2 Median 5 3 3 1 7 7 4 5 4.5 3 5 5 2 5.5 4 Mode 6 5 1 1 7 7 6 5 4 3 5 1 2 6 4

3.4. System Usability Scores (SUS)

At the end of each testing session, we gave users a post-test questionnaire. We used system usability scale (SUS) to compared the usability of each website. OneTravel has an average score of all users is 55.3±21.8 in while Hipmunk has 51.8 ± 21.2. Figure 7 shows SUS score of each user. These average scores tell us that users like OneTravel than Hipmunk in general since OneTravel get higher average SUS score. Despite the longer time take for user to complete the tasks; OneTravel received a higher Average System Usability Score from the users.

Table 3 lists all SUS scores from users and their first websites in this test.

Results show that 7 out of 10 users do not like the website that they began with in this test. This can be inferred that they have to accustom to all tasks and try to learn the process. Once they finished first set, they have a big picture in their mind. Therefore,

0!

2!

4!

6!

8!



Compare!Prices!


Bme!


Average�

Task�

Average!Score!for!Onetravel�

Q1!

Q2!

Q3!

!

10!!

when begins with second website, they can handle, feel easier and more satisfied with tasks.

Figure 7. SUS score of each user for OneTravel and Hipmunk

Table 3.SUS scores of each participant.

OneTravel Hipmunk First Website

P1 32.5 62.5 OneTravel

P2 52.5 25 Hipmunk

P3 62.5 82.5 OneTravel

P4 42.5 55 Hipmunk

P5 35 32.5 OneTravel

P6 45 42.5 OneTravel

P7 37.5 70 OneTravel

P8 65 80 OneTravel

P9 100 30 Hipmunk

P10 80 37.5 Hipmunk

Average 55.3 51.8

STD ±21.8 ±21.2

P1! P2! P3! P4! P5! P6! P7! P8! P9! P10!OneTravel! 32.5! 52.5! 62.5! 42.5! 35! 45! 37.5! 65! 100! 80!Hipmunk! 62.5! 25! 82.5! 55! 32.5! 42.5! 70! 80! 30! 37.5!

0!

20!

40!

60!

80!

100!

SUS!!Score�

SUS!Score!of!ParFcipants!

!

11!!

4. Usability Testing Results

We have classified the usability testing results based on the web pages they were found. The three different webpages of OneTravel are as following:

• Homepage • Loading Page • Search Result Page

4.1. Homepage

Finding 1: The majority of the participants do not notice the "Vacation” booking

Problem Description The label “Vacation” is not clear enough for the user to recognized. This is one of the reasons that the user never actually notice the availability of this funciton.

Severity 4

Recommendation for Improvement

Make the label more obvious e.g. use Flight + Hotel as the label or give a pop up description when user put the mouse over the tab

Relation to the heuristic result

Confirmed with the heuristic finding.

Encountered by participants

1,2,3,4,5,6,8,9,10

Screenshot

!

12!!

Finding 2: Search tab reset to “Flight” every time the user refreshes or comeback to the homepage from the previous page.

Problem Description Every time the user have to come back to the homepage to search again, the tab position will reset to “Flight”. This becomes troublesome especially when user is doing a vacation booking. Due to a similar search parameters many users did not even recognize that they are searching in a wrong type of booking.

Severity 2


Remember the user’s last used preference.


This is a new finding as compared with the heuristics evaluation.


1,2,4,5

Screenshot

Finding 3: OneTravel only allows Single Session.

Problem Description When an expert user would like to do a multiple search, it is likely that they would open search in different tabs. However, OneTravel do not allow this utilization of the website.

Severity 2


Provide multiple sessions to a single user.




!

13!!

Screenshot

4.2. Loading page

Finding 4: Load Page Progress is not visible

Problem Description The progress indicator, dots at the bottom of the page, is not visible to the user. User may think that the system stopped or get confused that they are in the wrong place and go back to home page just to find out that there is nothing wrong.

Severity 3


Add animation to show progress.




1,3,4,5,9,10

Screenshot

!

14!!

4.3. Search result page

Finding 5: Confusing 4 steps task separation

Problem Description Booking steps are confusing (the user do not know that their flight is pre-selected, and have trouble finding a way to change the flight)

Severity 2


Make it more clearly for the user to change flight.


Lowered the severity rating from 4 to 2.


1,2,3,10

Screenshot

Finding 6: Page Information Overload

Problem Description There is a lot of information in the page. This causes the user to not be able to find many different important elements within the page. For example, the user can not find the “TripAdvisor” score, can’t find show map button, can't find the Change Flight link)

Severity 3


Change the layout to make room for a clutter-free interface with focus on new and specific information.


Increased the severity from 2 to 3. The users were totally lost in search of tripAdvisor rating, it’s meaning, change flight button after selecting the hotel.


1,6

!

15!!

Screenshot

Finding 7 Checkout button is not clearly visible

Problem Description Checkout button is not clearly visible and lost among the less important information. The user needs to search for it after making his/her choices.

Severity 1


Make the button stand out from the other information present on the page.


Lowered the severity rating from 4, in heuristic evaluation, to 1. The users managed to search the checkout button.


10

Screenshot

!

16!!

Finding 8 Unclickable Breadcrumb

Problem Description Unclickable Breadcrumbs on the search result page, making it hard for the users to move backward/forward between each stem

Severity 2


Provide clickable breadcrumbs to the user to ease out their forward/backward movements.


Lowered the severity rating from 3, in heuristic evaluation, to 2. The users used the browser back button all the time to go back to the previous page.


1,4

Screenshot

Finding 9 Browser Back Navigation does not work properly

Problem Description When pressing the back button, the user will be taken back to the loading page not the page before. For example if they have selected hotel and then select flight. From the flight page if they press back they will go back to loading page instead of hotel.

Severity 2


Allow user to go back each step using this browser back button.


This is a new finding as compared with the heuristics evaluation. The experts didn’t account for the backward/forward movement of the website in the heuristic evaluation.


1,3,4

Screenshot

!

17!!

Finding 10 Confusion between the hotel’s Star Rating and TripAdvisor

Problem Description There is a serious confusion between the hotel’s Star Rating and TripAdvisor user review ratings. Even with the users who already understand the different between them. This is caused by the inconsistency of the availability of filter on the website (only for Star-rating)

Severity 4


Clearly differentiate between the Star Rating and TripAdvisor rating and put a legend describing them.




1,3,4,8, 9,10

Screenshot

Finding 10 No Multiple Hotels map view

Problem Description User cannot view all hotels in the single map. This makes it very difficult for the user to compare location of each hotel in the result. If they would like to compare each hotel then they have to print the map out and then mark the location and price of interested hotels.

Severity 4


Add a single map to illustrate the hotel’s location.




1,2,3,4,5,6,9,10

!

18!!

Screenshot

Finding 11 Difficulties in comparing hotels with 2 or more criterion

Problem Description OneTravel support the sort and filtering result with a single criterion. However, not every users desire to find the cheapest hotel or the most popular hotel. User can also want to select a hotel with 2 criterion, price and users’ rating. In this case, OneTravel cannot support this need.

Severity 4


Add a filter or secondary sort function.


This is a new finding as compared with the heuristics evaluation. The experts compared the hotels based on the given filters only.


1,2,4,6,7,9,10

Screenshot

!

19!!

Finding 12 Unclear label for Selected Search Date, Destination in the search result page

Finding 13 Sort by requested time does not sort by requested time !

Problem Description Sort by requested time does not sort by requested time in the search result page. There are instances when early departure flights are shown way down the list.

Severity 4


Make the sort by requested time option more powerful.




1,4,5

Problem Description Unclear label for Selected Search Date, Destination in the search result page.

Severity 2


Make the Search Date more prominent




1,3,4

Screenshot

!

20!!

Screenshot

Finding 14 Unrequested Search Result !

Problem Description The result page gives the Near-by airport result at the top and not the actual search queried by the user.

Severity 4


Only show result from selected airport, or give option for user to hide near-by airport.


Increased the severity rating from 2 to 4. The users made the mistakes of selecting the first flight shown in the search result page.


1,2,3,4,5,6,7,8,9,10

Screenshot

!

21!!

5. Synopsis

5.1. Usability Problems

The findings of the user testing can be summarized in to 3 aspects that OneTravel underperformed when compare to Hipmunk: Finding information in the website, Comparing hotels and Comparing flights.

• Finding information in the website

OneTravel has serious problems with how important information is placed in the website and what label is used to hint the user of the function. 9 out of 10 users never used the Vacation booking option due to these two reasons. There are too many competing pieces of information, which make the user unable to make sense of and/or choose to ignore. In addition, “Vacation” is not an adequate label for the functions. Users only notice the meaning of vacation by process of elimination of the 4 available tabs. However they never realize this without being asked by the facilitator.

• Comparing Hotels with multiple constraints or location constraint OneTravel perform exceptionally well in the case when user would like to get the cheapest, most popular or best ranking hotel. However when the user have more than one decision criteria such as, wanting to get the cheapest hotel which is still reasonably good quality within its price range. User will then look at both price and TripAdvisor review, however, with the current functionality it is impossible achieve this without having to compare each hotel one by one. There is another major problem when the users want to compare a hotels using location, e.g. want the hotel to be close to downtown. This can be said to be impossible without using other tools. Many users try to use Google Maps to help locate the hotels and compare them individually. In real situation, users are more likely to entirely change the booking website and this is the sign of failure to support users’ goal.

• Comparing Flight: Unrequested information and Time Selection Similarly to hotel comparison, there is a major usability issue within flight comparison process. The problem is mainly caused by OneTravel providing unrequested information such as Near-by airport or other airlines even the user specifically select one airline. Instead of providing a flexibility of choice, this prevents users from find the desired flight, let alone comparing them.

!

22!!

Another relating problem is when the user would like to book flight that has a particular time, for example at 10am for both depart and return flight. There is no functionality to support this comparing process. The user will have look at every result in order to find the desire flight. OneTravel also force user to choose depart and return flight at the same time, which make the process even more difficult.

5.2. Recommendation

Despite the lower success rate, longer time to finish the task, higher perceived difficulties and lower System Usability Score (SUS). Many users, in the final interview, prefers OneTravel due to then conventional style of e-commerce website. However, many users also notice the ease of use and the ease of learning in Hipmunk. Without any prompt action to these serious issues might cause the user to use the competitor service instead. Here are the general guidelines about the strength, something to maintain in the new design, and

OneTravel Strength

• Conventional e-commerce style: Most users feel at ease when using OneTravel because they can predict what will happen and therefore feel safe when using this website. Most users prefers the conventional style therefore when redesigning OneTravel can reconsider highlighting important functions and removing unnecessary information instead of redesigning the entire website. In addition, novels approach will be more likely to be accepted by tech savvy users only.

• Allowing Manipulation: Users feel more confident when they can manipulate

the result before finding the desire ticket or hotel rather than being pre-selected by the system.

• Single Payment: Users agree that buying hotel and ticket together is easier and more convenient.

OneTravel Needs for Improvement

• Information Architecture and Labeling: OneTravel can be improve immensely with a better arrangement of information, for example highlighting or emphasize on important information. This can eliminate the possible confusion that might occurs. Appropriate label will help guiding the users.

• Better Cooperation of Functionality: There are many different functionality

provided by the OneTravel. However, many times the users are loss of how they can be used together to achieve their goals. OneTravel have to consider what the users’ goals are and how can they easily understand and use the functionality provided.

!

23!!

• Removing unnecessary information: Extra information can be useful to some users; however, this might not always be true. By providing an option for users to show extra information when needed, and only show information that the user deliberately ask for will help users find the information that they want.

!

24!!

6. Appendix

6.1. User Testing Package

Usability Testing Script

Usability Script Preparation

1 Open Chrome in Private Browsing Mode (CMD + Shift + N) 2 Open OneTravel or Hipmunk depending on the order 3 Open Silverback Software (Recording should start when you start saying the introduction 4 Minimize all the Webpages

Activity Outline

1 Start Introduction Script 2 Scenario 1 Website A 3 Scenario 2 Website A 4 SUS Questionnaire 5 <Repeat Scenario step 2 - 5 for Website B> 6 Final Interview

1. Introduction <Start Silverback Recording> Welcome

● Thank you for agreeing to help us! My name is _______________, we are students in the school of informatics and this work is for our class project.

● Today, we would like you to help us evaluate on an existing website and give us some feedbacks.

Encourage Comments ● Before we move on, I would like to stress that we are not testing you or your ability to complete

the things that we ask you to do. ● We’re looking for your honest feedbacks. ● We would like to see how people use it. If you do not know what to do or the website does not

work as you expected, that is the best kind of feedback you can give. Brief Background

● I am going to give you some task to complete with the website that I am going to show you. ● I would like you to “think out loud” as you try things. Say what you are looking at, what you are

trying to do or what you are thinking. ● For example, if something doesn’t work the way you expected you might say “Hmm that’s not

what I expected; I thought it was going to do this...” Warm up questions

● Your name is _______, right? ● What is your major? ● What do you think of the weather in Indianapolis?

!

25!!

Now we would like to talk about a online booking website. Have you ever use any online booking website?

2. Scenario 1 <Open the website A> Now, let’s say that you know you are going to have a holiday in March. You would like book a trip to New York. But before we start I would like you take a minute or two to look at the website and get familiar with it. Please tell me when you are ready. <Allow 1-2 minutes for them to look at the website> As you are completing this please try to “think out loud” as much as possible. Please start. Task 1.1: You want to book a trip to New York from 5th March 2013 to 8th March 2013. You need both air ticket (for both departure and return) and a hotel. You will be leaving from Indianapolis International Airport (IND) and be going to Kennedy International Airport, New York City (JFK). Find the cheapest direct air ticket and find the cheapest hotel with a User’s rating of at least 4 points. There are 2 different ratings in the website, Star Rating and User Rating. Star Rating is used to classify different classes of hotels, for example, 4-star is for First Class hotels, 5-star is for Luxurious hotels. However, User rating is the points that user gives to the hotel rating their experience, such as 1 points if they are not satisfied or 5 points if they are satisfied. <If they start booking using flight and hotel separately redirect them to vacation. > You have done great so far. Now I would like to redirect your attention to this page. <Redirect them to the Vacation Booking> This provides an option for people to book air ticket and hotel together. Please continue to book the same trip using this function. <Post-Task Questionnaire> Task 1.2: Now that you have selected all desired air ticket and hotel, find out how you will continue to pay for the trip. <Post-Task Questionnaire> Task 1.3: As you are about to pay for your trip, you remember that you do not have to leave on the 5th March but you can also leave on the 6th and the 7th March. Compare the price of the current trip, the trip from 6th and return on the 9th March and the trip from the 7th and return on the 10th March

!

26!!

<Post-Task Questionnaire>

Now you can take a small break, please tell me when you are ready to move on :D

3. Scenario 2 So this time, let’s say you are booking for your parent. They will go to Atlanta for 3 days. Since they are older there have certain preference for the flight and the hotel. Book an air ticket and a hotel with the following requirements. Task 2.1 Find any flight (round-trip) accordingly in these criterions.

● Date: 5th - 8th March ● Depart from Indianapolis International Airport (IND) to Atlanta Hartfield-Jackson

International Airport (ATL) ● For both departure and return flight, the plane should leave between 10am - 6pm so that they

do not have to wake up early or arrive too late at night <Post-Task Questionnaire> Task 2.2 Find any hotel according to these criterions.

● 4-star hotels ● Hotel should locate in downtown Atlanta or the closest to downtown.

<Post-Task Questionnaire> 4. SUS Questionnaire 5. Interview Now you can take a small break, please tell me when you are ready to move on :D

6. Repeat Step 2 – 5 for Website B <Open the website B> Great. Now I would like you to do this scenario for this website. Like the last website, I would like you take a minute or two to look at the website and get familiar with it. Please tell me when you are ready. <Allow 1-2 minutes for them to look at the website>

7. Final Interview

!

27!!

Pre-Test Questionnaire

Website:___________________)))))))))))))))))))))))Participant)Number:)______________________________)

!

Page!1!

Pre-Test Questionnaire

Name: _______________________________

Sex: □ Male □ Female

Age: □18-24 □25-30 □30-41 □Over 41

Occupation: (if a student, please provide your field of study) __________________________________

1. How long do you usually use Internet, for both web browsing and email in a week?

□ Less than 1 hour □ 1 – 3 hours □ 4 – 5 hours □ 6 hours or more

2. How would you rate your expertise using Online Travel Reservation website?

□ No Experience □ Beginner □ Competent □ Expert

3. In the past six months, how many times have you booked a flight or a hotel?

□ 0 □ 1 □ 2 □ 3 □ 4 □ 5 or more

4. In the past six months, how many times have you used Online Travel Reservation website?

□ 0 □ 1 □ 2 □ 3 □ 4 □ 5 or more

5. What are the Online Travel Reservation websites that you have used before ? (Select all that apply)

□ OneTravel.com □ Expedia.com □ Kayak.com □ Hipmunk.com

□ Other (Please specify) __________________________________________________________

!

28!!

Task Sheet

!

29!!

Post-Task Questionnaire

Scenario)2)So this time, let’s say you are booking for your parent. They will go to Atlanta for 3 days. Since they are older there have certain preference for the flight and the hotel. Book an air ticket and a hotel with the following requirements. Task%No.%

Task%Description%

221% Find a flight according to these criterions.

• 5th – 8th March • Depart from Indianapolis International Airport (IND) to

Atlanta Hartfield-Jackson International Airport (ATL) • For both departure and return flight, the plane should leave

between 10am - 4pm so that they do not have to wake up early or arrive too late at night

222% Find a hotel according to these criterions.

• 4-star hotels • Hotel should located in downtown Atlanta

)


!

Page!2!

Post-Task Questionnaire

Task Number: _______________________________

1. How was the overall task? Very Easy Neutral Very

Difficult

2. I was satisfied with the ease of completing this task? Strongly Disagree

Neutral Strongly Agree

3. I was satisfied with the amount of time it took to complete the task? Strongly Disagree


!

30!!

System Usability Score (SUS) Questionnaire


!

Page!5!

SUS Questionnaire

1. I think that I would like to use this website frequently. Strongly Disagree


2. I found this website unnecessarily complex. Strongly Disagree


3. I thought the system was easy to use. Strongly Disagree


4. I think that I would need the support of a technical person to be able to use this system. Strongly Disagree


5. I found the various functions in this website were well integrated. Strongly Disagree


6. I thought there was too much inconsistency in this website. Strongly Disagree


7. I would imagine that most people would learn to use this website very quickly. Strongly Disagree


8. I found this website very complicated to use. Strongly Disagree


9. I felt very confident using this website. Strongly Disagree


10. I needed to learn a lot of things before I could get going with this website. Strongly Disagree


!

31!!

6.2. Quantitative Data

Success Rate Data

Participant

One Travel



Find the how to pay

Find and Compare Flight



P1 0 1 1 0 0 P2 0 1 0 1 1 P3 0 1 0 0 0 P4 0 1 1 1 1 P5 0 1 1 1 1 P6 1 1 1 1 1 P7 1 1 1 1 1 P8 1 1 1 1 1 P9 1 1 1 1 1

P10 1 1 0 1 1 Success 5 10 7 8 8

Success rate (%) 50 100 70 80 80

Participant

Hipmunk



Find the how to pay




P1 1 1 0 1 1 P2 1 1 1 1 1 P3 1 1 1 1 1 P4 1 0 1 1 1 P5 1 1 1 1 1 P6 1 1 1 1 1 P7 1 1 1 1 1 P8 1 1 1 1 1 P9 1 1 1 1 1

P10 1 1 1 1 1 Success 10 9 9 10 10

Success rate (%) 100 90 90 100 100

!

32!!

Time-on-Task Data

Participant

Time-on-Task in One Travel (min)



Find the how to pay




P1 8.7 1.5 2.0 7.9 7.5 P2 2.5 0.9 3.5 5.4 1.6 P3 5.3 0.8 2.9 4.6 3.3 P4 6.4 1.2 8.0 1.6 0.0 P5 8.4 0.1 3.0 0.9 1.3 P6 17.4 1.0 9.5 10.2 6.1 P7 5.1 0.1 6.2 11.1 9.0 P8 12.6 0.3 11.6 5.5 5.4 P9 6.7 1.3 5.3 3.1 4.0 P10 11.8 1.9 7.9 6.7 10.3 Min 2.5 0.1 2.0 0.9 0.0 Max 17.4 1.9 11.6 11.1 10.3 Average 8.5 0.9 6.0 5.7 4.8 Std 4.4 0.6 3.2 3.4 3.4

Participant

Hipmunk Time-on-Task in Hipmunk (min)



Find the how to pay




P1 3.8 2.2 3.5 2.7 1.3 P2 3.7 3.2 2.3 2.7 0.6 P3 5.0 2.2 4.3 1.2 2.1 P4 5.9 0.5 5.6 1.6 1.9 P5 4.0 1.3 3.0 1.3 2.1 P6 13.5 1.0 10.2 10.1 6.6 P7 3.0 0.2 5.0 3.0 5.0 P8 9.8 0.3 12.0 2.6 4.0 P9 4.9 0.5 6.1 4.6 3.1 P10 7.0 3.3 4.7 1.4 2.5 Min 3.0 0.2 2.3 1.2 0.6 Max 13.5 3.3 12.0 10.1 6.6 Average 6.1 1.5 5.6 3.1 2.9 Std 3.3 1.2 3.1 2.7 1.8

!

33!!

Post-Task Questionnaire Data

Q1 How was the overall task? (1 very easy - 7 Very Difficult)

Q2 I was satisfied with the ease of completing this task. (1 Strongly Disagree - 7 Strongly Agree)

Q3 I was satisfied with the amount of time it took to complete the task (1 Strongly Disagree - 7 Strongly Agree)

!

34!!

Participant

One Travel Post-Task Questionnaire

Task 1 Task 2 Task 3 Task 4 Task 5

Find Flight and Hotel Find the how to pay Find and Compare Flight



Q1 Q2 Q3 Q1 Q2 Q3 Q1 Q2 Q3 Q1 Q2 Q3 Q1 Q2 Q3 P1 6 3 3 4 5 5 6 6 6 7 1 1 6 2 1 P2 6 2 3 1 5 5 4 5 4 7 1 1 2 4 4 P3 5 5 7 1 7 7 6 2 4 3 2 7 7 6 4 P4 3 1 1 2 5 4 4 5 4 3 5 5 1 7 7 P5 5 3 2 1 7 7 2 5 2 1 7 7 6 3 2 P6 6 2 1 1 7 7 5 2 5 1 5 1 2 6 4 P7 6 1 1 1 7 7 3 3 2 2 6 6 2 5 6 P8 2 5 5 1 7 7 5 4 5 3 5 5 1 6 6 P9 4 4 4 4 7 7 1 7 7 4 4 4 7 1 1 P10 4 5 5 4 3 3 3 5 6 1 7 7 1 7 7 AVG 4.7 3.1 3.2 2 6 5.9 3.9 4.4 4.5 3.2 4.3 4.4 3.5 4.7 4.2 MEDIAN 5 3 3 1 7 7 4 5 4.5 3 5 5 2 5.5 4 MODE 6 5 1 1 7 7 6 5 4 3 5 1 2 6 4

!

35!!

Participant

Hipmunk Post-Task Questionnaire

Task 1 Task 2 Task 3 Task 4 Task 5

Find Flight and Hotel Find the how to pay Find and Compare Flight Find Flight w/ specific time Find Hotel w/ specific location

Q1 Q2 Q3 Q1 Q2 Q3 Q1 Q2 Q3 Q1 Q2 Q3 Q1 Q2 Q3 P1 2 6 6 4 4 3 3 5 3 2 6 5 3 5 6 P2 5 5 4 7 4 4 2 6 7 4 3 3 1 6 7 P3 7 2 4 7 1 3 2 6 6 1 7 7 3 6 6 P4 4 4 3 2 2 2 5 4 3 2 6 6 3 6 6 P5 5 2 6 7 1 1 5 4 3 2 5 5 2 5 5 P6 4 5 5 5 4 3 3 5 7 2 7 3 3 5 6 P7 4 3 5 7 1 1 1 3 4 2 3 3 3 1 2 P8 5 2 4 5 3 2 2 6 7 1 7 7 1 7 7 P9 7 1 1 4 4 4 1 1 1 1 1 1 7 7 7 P10 3 5 7 2 7 7 2 7 7 1 7 7 2 6 6 AVG 4.6 3.5 4.5 5 3.1 3 2.6 4.7 4.8 1.8 5.2 4.7 2.8 5.4 5.8 MEDIAN 4.5 3.5 4.5 5 3.5 3 2 5 5 2 6 5 3 6 6 MODE 5 5 4 7 4 3 2 6 7 2 7 3 3 6 6

onetravel final report onetravel

Documents

usability testing results

testing procedure

user testing package

usability problems

accumulated success

system usability scores

task conditions

user demographics