user-centered research on the paying for college website and tools - edui 2014

Paying For Education: User-Centered Research on the Paying For College Website and Tools September 30, 2014 Jennifer Romano Bergstrom & Lamar Alayoubi @romanocog

Eye Tracking in UX – Oct 16

2

User Focus – Oct 17 eyetrackingux2014.com October 16, 2014 EDUI = 20% OFF

2014.userfocus.org October 17, 2014 EDUI = 20% OFF

Usability = “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use.” ISO 9241-11

+ emotions and perceptions = UX

@romanocog #eduiconf

4 User Experience Design (P. Morville): http://semanticstudios.com/publications/semantics/000029.php

User Experience


5

When to test


6

What we design for…and reality

Krug, S. (2000) Don’t Make Me Think. Pearson Education *NEW 3rd Edition (2014)*

@romanocog #eduiconf @skrug

7

OBSERVATIONAL + Time on page/task

+ Reaction time + Selection/click behavior

+ Success/fail rate + Conversion rate

IMPLICIT + Eye tracking

+ Electrodermal activity (EDA) + Behavioral analysis

+ Verbalization analysis + Pupil dilation

+ Facial expression analysis

SUBJECTIVE + Satisfaction & difficulty ratings

+ Verbal responses + Moderator follow up

+ Real-time +/- dial

UX Data @romanocog #eduiconf

Traditional UX research is good at explaining what people say and do, not what they think and feel.

Why should we measure implicit?


9 User Experience Design (P. Morville): http://semanticstudios.com/publications/semantics/000029.php

@romanocog @richmondux

Implicit Data


10

Modern eye tracking @romanocog @dcaapor

Eye Tracking in the Past

11

Modern eye tracking

Modern Eye Tracking

UX Research on Paying For College

12

Tested the User Experience (UX) of: •  Overall navigation •  Comparing the cost of colleges side-by-side •  Organization of information:

•  Federal vs. private loan information •  Federal loan interest rates •  Financial aid disbursement

information •  College bank accounts.

Task Creation

SATQ Items

Other materials

Debriefing

Reports

.

. .

.

. .

.

Iterative Process with Client


“A spectator sport” @skug

Testing Iterations & Rounds

13

High School Students

Influencers

Round 1: Austin, TX

Round 3: Arlington, VA

Round 2: Spokane, WA

Live site version 1 Live site version 1 Live site version 1 OR Prototype site version 2

Iteration 1:

February/ March 2014


College Graduates

Round 1: Atlanta, GA


Round 2: Milwaukee, WI

Live site version 2 Live site version 2 Live site version 2

Iteration 2:

April 2014


14

Methodology @romanocog #eduiconf

Testing Environment

Eye Tracker

Moderator & Participant

Methods: UX Metrics

15

OBSERVATIONAL + Time to complete tasks

+ Time to first click + First click accuracy

+ Task accuracy


SUBJECTIVE + Difficulty and satisfaction ratings

+ Think aloud protocol + Debriefing interview

•  Eye tracking [implicit] tells us what participants focus on. •  Think aloud & debriefing [subjective] tell us why they focus there. •  Observational measures tell us how they go about completing a task. •  Combination of subjective, observational, & implicit metrics paints the whole picture.

Use eye tracking to: •  Gain insights into attracting visual elements and the order that users observe •  Identify the amount of text that respondents read (e.g., terms/instructions) •  Classify how respondents complete tasks and the order in which they process elements


UX Metrics in Action: Live Website Testing

16


Navigation should be intuitive.

TASK: Let’s say that you have your list narrowed down to the two schools that you have financial aid offers from. You are interested in graduation rates of those two schools. Which university graduates a higher percentage of students who enroll?

17

First clicks: Compare financial aid offers (14),

Compare costs (3), search function (1)

17

1

Successful

Iteration 2 Measure: First-click accuracy

Unsuccessful

6 6 Successful

Unsuccessful

First clicks: Compare costs (4),

Research schools (3), Compare get started (2),

Search function (1), Home (1), Graduate (1)


50% Accuracy:

94% Accuracy:


Navigation should be intuitive.


18

First clicks: Compare financial aid offers (14),

Compare costs (3), search function (1)

17

1

Successful


Unsuccessful

6 6 Successful

Unsuccessful

First clicks: Compare costs (4),

Research schools (3), Compare get started (2),

Search function (1), Home (1), Graduate (1)


50% Accuracy:

94% Accuracy:

UX Best Practices & Recommendations:

•  Ensure that the primary navigation and other navigation options are intuitive and match user expectations.

UX Metrics:

•  First Click Accuracy: If people make the correct first click, the design is intuitive (for that particular task).


13

3

Successful

Unsuccessful

First clicks: Student loans (7),

More about student loans (5), Student financial guides (1),

Get Assistance – Info for students (1), Participate – Student loans (1), Compare financial aid offers (1)

11

1 Unsuccessful

Successful

First clicks: Choose a loan - timeline (8),

Choose a loan – Get started (3), Home (1)

Iteration 1 Measure: First-click accuracy. Task: “What are the types of loans available?”


94% Accuracy:

81% Accuracy:

Navigation should be intuitive and consistent. @romanocog #eduiconf

TASK: Let’s say you have decided on the university you want to attend and you know you’re going to need to take out a loan to pay for tuition. What are the different types of loans that are available to you?

13

3

Successful

Unsuccessful

First clicks: Student loans (7),

More about student loans (5), Student financial guides (1),

Get Assistance – Info for students (1), Participate – Student loans (1), Compare financial aid offers (1)

11

1 Unsuccessful

Successful

First clicks: Choose a loan - timeline (8),

Choose a loan – Get started (3), Home (1)



94% Accuracy:

81% Accuracy:

Navigation should be intuitive and consistent.


•  Ensure that navigation options are consistent; for a horizontal navigational layout, either all headings should have dropdown options, or none should.

•  Use category headings that accurately convey what users will find on those pages.


TASK: Let’s say you have decided on the university you want to attend and you know you’re going to need to take out a loan to pay for tuition. What are the different types of loans that are available to you?

Getting started with a tool should be intuitive.

21

Iteration 1: Gaze plots for 4 participants on Compare Costs tool main page. Disorderly gaze patterns suggest that it isn’t readily apparent where to start on this page.

Iteration 2: Gaze plots for 4 participants on Compare Costs tool main page. More orderly gaze patterns clustered at the top of the page suggest that it is more apparent where to start on this page.

27 sec Time to start:




Getting started with a tool should be intuitive.

22

Iteration 1: Gaze plots for 4 participants on Compare Costs tool main page. Disorderly gaze patterns suggest that it isn’t readily apparent where to start on this page.

Iteration 2: Gaze plots for 4 participants on Compare Costs tool main page. More orderly gaze patterns clustered at the top of the page suggest that it is more apparent where to start on this page.




•  It should be apparent to the user how to get started using a web tool.

•  Use a prominent location and, if necessary, highlight the button that users need to get started.

UX Metrics:

•  Time to First Click: If people make the correct first click quickly, the design is intuitive (for that particular task).

•  Eye Tracking Gaze Plots: Gaze patterns inform us of the order in which a user processes

information. Disorderly gaze patters imply a poor design.



23

Input methods should be apparent. •  When attempting to put a number in a box that

requires a previous box to be completed, no indication is given that participants have to fill out the prior information first, resulting in participants becoming increasingly frustrated.

•  Participants commented that there weren’t enough boxes to put in all of the information they had; specifically when entering scholarship information, there is only one blank for “other scholarships and grants,” prompting several participants to say that they wished there were additional blanks.

•  The Cost of Attendance dropdown automatically opens, but the Money for School dropdown does not. As a result, some participants did not notice that they could enter additional information, resulting in confusion.

“It made me think that the site was incomplete or I missed something.” “I don’t understand. What’s the point if you can’t compare the costs. Maybe I did it wrong.”

Iteration 1: Compare tool input fields.




24

Input methods should be apparent. •  When attempting to put a number in a box that

requires a previous box to be completed, no indication is given that participants have to fill out the prior information first, resulting in participants becoming increasingly frustrated.

•  Participants commented that there weren’t enough boxes to put in all of the information they had; specifically when entering scholarship information, there is only one blank for “other scholarships and grants,” prompting several participants to say that they wished there were additional blanks.

•  The Cost of Attendance dropdown automatically opens, but the Money for School dropdown does not. As a result, some participants did not notice that they could enter additional information, resulting in confusion.

“It made me think that the site was incomplete or I missed something.” “I don’t understand. What’s the point if you can’t compare the costs. Maybe I did it wrong.”




•  Provide relevant error messages when users enter information in the incorrect order (or skip a field) to promote recovery and minimize frustrations.

•  Provide enough places to enter information – match user expectations. In this particular example, more options for entering scholarship amounts into the tool would have been clear and less cognitively taxing for the user.

UX Metrics:

•  Think Aloud Protocol: Users verbalize their experience while completing tasks and tell us in-the-moment when they are frustrated. We quantify the percentage of users that verbalize a particular problem and gain insight into how severe the problem is.



Infographics should be easily understandable.

25

•  Because participants are confused about how to enter their information into the tool correctly, the output graphics that the tool generates are also confusing.

•  The Left to pay number is not calculated into the debt burden, leading to several participants being confused as to why they’ll have so little debt at graduation.

•  No explanation is provided for where salary is pulled from, leading several participants to question the accuracy of that number.

Iteration 2: Compare tool output graphics.


TASKS: After you graduate, what would be the debt burden for [THE SECOND SCHOOL THEY ENTER]? After you graduate, what is the monthly payment for the loans you would take out while attending [THE FIRST SCHOOL THEY ENTER]? How much debt burden would you have at [SECOND SCHOOL] vs. [FIRST SCHOOL]?

Infographics should be easily understandable.

26

•  Because participants are confused about how to enter their information into the tool correctly, the output graphics that the tool generates are also confusing.

•  The Left to pay number is not calculated into the debt burden, leading to several participants being confused as to why they’ll have so little debt at graduation.

•  No explanation is provided for where salary is pulled from, leading several participants to question the accuracy of that number.

Iteration 2: Compare tool output graphics.


•  Thoroughly explain how to use a diagram; do not assume that the user will be able to figure out how to interpret an infographic.

•  Use hover-over pop-up boxes to explain what each portion of the infographic actually represents.

•  Identify sources of information whenever appropriate. For example, salary information and graduation rates should be explained and sourced in this infographic.


TASKS: After you graduate, what would be the debt burden for [THE SECOND SCHOOL THEY ENTER]? After you graduate, what is the monthly payment for the loans you would take out while attending [THE FIRST SCHOOL THEY ENTER]? How much debt burden would you have at [SECOND SCHOOL] vs. [FIRST SCHOOL]?

Users concentrate on the most important site features.

27

Participants in the Graduate group were asked what content on the site was relevant to them. They looked at the primary features of the site in order to locate relevant content.

PFC main homepage.

Fixation count heat map of 12 participants on the main page, fixating mostly on the PFC header and on the four main menu tabs, including the Repay Student Debt tab.


Users concentrate on the most important site features.

28

Participants in the Graduate group were asked what content on the site was relevant to them. They looked at the primary features of the site in order to locate relevant content.

PFC main homepage.

Fixation count heat map of 12 participants on the main page, fixating mostly on the PFC header and on the four main menu tabs, including the Repay Student Debt tab.

UX Metrics:

•  Eye Tracking Heat Maps: Heat maps allow us to see concentrations of people’s fixations; in this instance, people fixate the introductory text and the 4 labels of the main navigation, which is exactly where we want them to look.


UX Metrics in Action: Prototype Testing

29


Repay Student Debt Prototype Methodology

30

*Prototypes presented were non-functional PDF prototypes of the starting page of the repayment tool.

CFPB proposed testing three non-functional prototypes of the main page of the Repay Student Debt tool.* The look and feel of those prototypes differed in each round of testing.

Borrower Type: Borrower type allowed participants to choose whether they were a typical borrower or had special circumstances. Version 1 of the Borrower prototype included images, while Version 2 did not. All participants saw one version of this prototype.

Loan Type: Loan type allowed participants to select what type(s) of loans they have. This was similar to how the live site worked. Version 1 of the Loan prototype included images, while Version 2 did not. All participants saw one version of this prototype.

Repayment Type: Repayment type allowed participants to choose what kind of repayment plan they were interested in. Only Round 2 participants saw this prototype.

Goal Type: Goal type allowed participants to choose whether they wanted to lower their monthly payment, pay less interest over time, or inquire about other programs. Only Round 3 participants saw this prototype. The main purpose of the prototype testing was to determine which starting point participants thought was the most logical. Four different starting points were tested:


Participants “clicked” the ‘Get Started’ content on each prototype first.

31

A majority of participants said that they would click on the Get Started content on every version of all prototypes except the repayment type prototype.

*Repayment Type prototype was only seen by Round 2 participants; the Goal Type prototype was only seen by Round 3 participants.

0

4

8

12

16

Borrower Type Loan Type Repayment Type*

Goal Type*

First “Clicks” on Any of the Get Started Buttons

Round 2 Round 3 Borrower Type Version 2.


Participants “clicked” the ‘Get Started’ content on each prototype first.

32

A majority of participants said that they would click on the Get Started content on every version of all prototypes except the repayment type prototype.

*Repayment Type prototype was only seen by Round 2 participants; the Goal Type prototype was only seen by Round 3 participants.

0

4

8

12

16

Borrower Type Loan Type Repayment Type*

Goal Type*

First “Clicks” on Any of the Get Started Buttons

Round 2 Round 3 Borrower Type Version 2.

UX Metrics:

•  First Clicks: PDF prototypes were not clickable. We ask participants where they would click first and gain insight into how users would interact with these pages if they were live.


Prototype Summary & Recommendations

33

•  There was no overwhelming favorite among participants regarding which prototype was their favorite. –  4 out of 18 (22%) participants preferred the Borrower Type prototype –  6 out of 18 (33%) participants preferred the Loan Type prototype –  4 out of 8 (50%) participants preferred the Repayment Type prototype –  4 out of 10 (40%) participants preferred the Goal Type prototype

Recommendations: Start with either the goal or the loan prototype. Although there was no real consensus around one prototype, almost all participants said something positive about the goal prototype. Having a goal-oriented way of getting into the tool matches how users think of the process. The loan prototype was basically a different-looking version of the live site, which participants found generally intuitive. Include a ‘Help Me Decide’ button. Help Me Decide was a popular first click button on the one prototype it was shown with. It would allow users who don’t necessarily have a goal a way to get in. Include graphics. Although graphics were removed from the second iteration of the prototypes for content-testing purposes, the feedback on them was generally positive in the first round of testing. Remove content below the fold. Most participants clicked on one of the Get Started options; however, some clicked on the other content below the fold. If the main purpose of this site is to get people to use the tool, then content below the fold could be removed to simplify the page.


Prototype Summary & Recommendations

34

•  There was no overwhelming favorite among participants regarding which prototype was their favorite. –  4 out of 18 (22%) participants preferred the Borrower Type prototype –  6 out of 18 (33%) participants preferred the Loan Type prototype –  4 out of 8 (50%) participants preferred the Repayment Type prototype –  4 out of 10 (40%) participants preferred the Goal Type prototype

Recommendations: Start with either the goal or the loan prototype. Although there was no real consensus around one prototype, almost all participants said something positive about the goal prototype. Having a goal-oriented way of getting into the tool matches how users think of the process. The loan prototype was basically a different-looking version of the live site, which participants found generally intuitive. Include a ‘Help Me Decide’ button. Help Me Decide was a popular first click button on the one prototype it was shown with. It would allow users who don’t necessarily have a goal a way to get in. Include graphics. Although graphics were removed from the second iteration of the prototypes for content-testing purposes, the feedback on them was generally positive in the first round of testing. Remove content below the fold. Most participants clicked on one of the Get Started options; however, some clicked on the other content below the fold. If the main purpose of this site is to get people to use the tool, then content below the fold could be removed to simplify the page.

UX Metrics:

•  Debriefing Interviews: After the tasks and testing is over, a debriefing interview allows us to ask users about their preferences. We can then quantify those preferences to gain insight into which prototypes users prefer.


•  Combination of subjective, observational, and implicit data to evaluate the UX. •  Many UX elements improved from Iteration 1 to Iteration 2; some need further exploration.

35

0%

0%

17%

56%

28%

0%

8%

8%

83%

0%

Extremely Difficult

Very Difficult

Moderately Difficult

Slightly Difficult

Not Difficult at All

0%

0%

6%

44%

50%

8%

0%

33%

58%

0%

Extremely Difficult

Very Difficult


Slightly Difficult


“Please rate your overall experience with the Paying for College website.”

“Please rate your difficulty with navigating on the Paying for College website.”

Iteration 1 Iteration 2

Study Impacts @romanocog #eduiconf

•  Combination of subjective, observational, and implicit data to evaluate the UX. •  Many UX elements improved from Iteration 1 to Iteration 2; some need further exploration.

36

0%

0%

17%

56%

28%

0%

8%

8%

83%

0%

Extremely Difficult

Very Difficult


Slightly Difficult


0%

0%

6%

44%

50%

8%

0%

33%

58%

0%

Extremely Difficult

Very Difficult


Slightly Difficult


“Please rate your overall experience with the Paying for College website.”

“Please rate your difficulty with navigating on the Paying for College website.”

Iteration 1 Iteration 2

Study Impacts

UX Metrics:

•  Satisfaction Questionnaire: After the tasks and debriefing, asking users to rate their satisfaction allows us to make comparisons in iterative testing.


UX Research Success: User Types

37


Influencers

Round 1: Austin, TX




Iteration 1:



College Graduates





Iteration 2:

April 2014


UX Research Success: Client Collaboration

38


Influencers

Round 1: Austin, TX




Iteration 1:



College Graduates





Iteration 2:

April 2014


Task Creation

SATQ Items

Other materials

Debriefing

Reports

.

. .

.

. .

.

Iterative Process with Client

“A spectator sport” @skug

UX Research Success: Multiple Data

39

OBSERVATIONAL + Time to complete tasks

+ Time to first click + First click accuracy

+ Task accuracy


SUBJECTIVE + Difficulty and satisfaction ratings

+ Think aloud protocol + Debriefing interview

•  Eye tracking [implicit] tells us what participants focus on. •  Think aloud & debriefing [subjective] tell us why they focus there. •  Observational measures tell us how they go about completing a task. •  Combination of subjective, observational, & implicit metrics paints the whole picture.


UX Research Success

40

•  All relevant user types participated. •  Students •  Graduates •  Educators •  Parents

•  All relevant parties were involved. •  Designers •  Developers •  Key stakeholders

•  Multiple data sources provided insight into the complete experience. •  Iterative process: Test, improve, test again

•  Ensure changes improve the UX. •  Ensure something else did not break.


Thank you! •  Twitter: @forsmarshgroup •  LinkedIn: http://www.linkedin.com/company/fors-marsh-group •  Blog: http://www.forsmarshgroup.com/categories/blog/

Jennifer Romano Bergstrom

@romanocog [email protected]

Lamar Alayoubi

[email protected]

Eye Tracking in UX – Oct 16

42

User Focus – Oct 17 eyetrackingux2014.com October 16, 2014 EDUI = 20% OFF

2014.userfocus.org October 17, 2014 EDUI = 20% OFF

user-centered research on the paying for college website and tools - edui 2014

Education

user experience ux

research schools

walive site version

wilive site version

orprototype site version

eye trackingsubjective

elements eye

ux metrics15