What Does it Mean for a System to be Useful? An Exploratory Study of Usefulness Craig M. MacDonald, Ph.D. Pratt Institute, School of Information & Library Science

Michael E. Atwood, Ph.D. Drexel University, College of Computing & Informatics

25 June 2014 Designing Interactive Systems | Vancouver, BC2

HCI has evolved

2

Web, mobile, and personal technologies have drastically altered how, where, and

why people use computers. (Bødker, 2006)

User Experience (UX), which treats human-

computer interaction as a cognitive-emotional process, is emerging as a new

paradigm for the field. (Bargas-Avila & Hornbæk, 2011)

"The 21st Century Concert Experience" by Al Case is licensed under CC BY-NC-ND 2.0

Usability – is it the only factor?

3

"Tricycle" by Aslak Raanes is licensed under CC BY 2.0

“If ease of use was the only valid criterion, people would stick to tricycles and never try bicycles.”

- Douglas Engelbart

Usability = the gold standard Usability research has dominated HCI over the

past 40 years. It has yielded tremendous insights into how to

design systems that are easy to use and easy to learn.

But, little time has been spent on determining

whether systems are actually useful.

4 Source(s): Hornbæk (2010); MacDonald & Atwood (2013)

Usefulness is not new… Designing useful systems has long been cited as one of

the primary goals of user-centered design.

5

…but it’s mostly a mystery Studies of usefulness are largely absent from

HCI research. There is no empirical evidence about the relationships

between usefulness and other factors. There is no widely accepted definition of

usefulness in HCI. When people talk about usefulness, we can’t be certain

they’re talking about the same thing. This research aimed to address these two areas by

(1) defining usefulness and (2) studying usefulness in a controlled experiment.

6

So, what is usefulness? Many different definitions or uses of the term

usefulness, each of which covered one or more of the following: 1.  The functions provided by the system.

e.g., “the functions people really need in their work” (Gould & Lewis, 1985)

2.  The tasks users are trying to complete. e.g., “the role of the technology in accomplishing the user’s relevant tasks” (Maryniak-Nelson & Caldwell, 1992)

3.  The goals users are trying to achieve. e.g., “whether system can be used to achieve some goal” (Nielsen, 1993)

4.  The context in which the system is being used. e.g., “how technology can fit into users' actual social and/ material environments” (Nardi, 1996)

7

A working definition: Usefulness is “the extent to which a system’s

functions allow users to complete a set of tasks and achieve specific goals in a particular context of use.”

This may sound similar; according to the ISO:

Usability is “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specific context of use.”

8

What is usability? There is surprisingly little consensus about a

precise definition of usability. (van Welie, van der Veer, & Eliëns, 1999)

But there is broad agreement that usability

refers to effectiveness, efficiency, and satisfaction. (Chen, Germain, & Rorissa, 2009; Hornbæk 2006; Hornbæk & Law, 2007; ISO 9241-11; Nielsen, 1993)

9

Our (imperfect) distinction Usefulness

a system’s appropriateness for a specific context

Usability

its effectiveness, efficiency, and satisfaction within that context

10

Method We designed and conducted an exploratory

experiment* to: 1.  examine the effects of context on usefulness;

and 2.  explore the relationships between usefulness

and three other UX attributes: usability, aesthetics, and enjoyment.

*Yes, a laboratory study of context seems like an oxymoron but we believe

it is a good starting point for studying usefulness.

11

What is context? Many ways of defining context; for this study,

context is defined simply as:

12

USER TASK

TOOL

ENVIRONMENT

Source(s): Suchman (1987); Bannon (1990); Schilit & Theimer (1994); Hutchins (1995); Kuutti (1995); Nardi (1996); ISO (1997); Dey, Abowd, & Salber (2001); Dourish (2004); Räsänen & Nyce (2006); Connolly, Chamberlain, & Phillips (2008)

(diagram adapted from Shackel, 1991)

Controlled: User The user dimension can depend on many

possible sub-dimensions. So, we tried to control for this factor by

limiting the study population to a group likely to share common interests, knowledge, and interaction styles/preferences.

13

Independent Variable: Task-Env Three written scenarios: one “no scenario” as a

control and two scenarios that differed along the task and environment dimensions:

14

Task-Environment 1 Task-Environment 2 Task:

EXPLORATION (action mode): find reputable resources with facts, pictures,

charts/graphs, etc., to include in presentation about Climate Change.

Task: RETRIEVAL (goal mode): find whether

the name of the first computer was “IPECAC” and whether it was invented in

1928 at Drexel University.

Environment: Early on a weekday morning In the library computer lab

Class assignment

Environment: Late on a weekday evening In a student dormitory room

A wager with a friend

Independent Variable: Tool

15

Educational information portals were chosen because they: 1.  offer a range of potential use

cases; 2.  are relatively easy to learn;

and 3.  are familiar to the study

population.

Dependent Variables Variable Scale Items Source

USEFULNESS 1.  The website provides the right functions. 2.  I am able to use the website to complete my task(s). 3.  I am able to use the website to fulfill my goal(s). 4.  The website fits my current situation. 5.  The website is useful.

Developed for this research

USABILITY 6.  The website is easy to use. 7.  I feel in control when I am using this website. 8.  The website requires little effort to use. 9.  Using the website is effective.

De Angeli, Hartmann, & Sutcliffe (2009)

AESTHETICS 10. Everything goes together on this website. 11. The color composition is attractive. 12. The layout appears professionally designed. 13. The layout is pleasantly varied.

Moshagen & Thielsch (2010)

ENJOYMENT 14.  I find using the website to be enjoyable. 15. The actual process of using the website is pleasant. 16.  I have fun using the website.

van Shaik & Ling (2011)

GOODNESS 17.  I judge the website to be: bad—good Hassenzahl (2004)

16

Study Design We used an incomplete repeated measures

design in which participants were exposed to all three levels of each variable exactly once.

17

Post-Study Questionnaire First, participants responded verbally to the

following question: What makes a website useful to you? What criteria

do you look for? Next, participants answered questions

regarding: Experience with the websites used in this study Knowledge of the Internet (iKnow) (Potosky, 2007)

Basic demographic information (age, gender) Background (major, class level, number of HCI

courses)

18

Participants 36 undergraduate students enrolled at Drexel

University in the Information Technology, Information Systems, or Software Engineering programs. All sophomores or above. Nearly all (35; 97%) had taken at least one HCI course.

Other characteristics: Median age range: 21 years old Gender: 86% male, 14% female iKnow (0-70): ranged from 52 to 70 (average = 62.81)

19

Data Analysis: Marginal Models A special case of Linear Mixed Models that

include: Fixed effects: can be interpreted similarly to

ANOVA Estimates of regression coefficients: can be

interpreted similarly to traditional regression analysis

20

Parameter Estimate t Sig.

USABILITY 1.386 16.834 < 0.001 **

AESTHETICS -0.633 -5.613 < 0.001 **

AESTHETICS * Climate Ch. 0.472 3.600 0.001 **

AESTHETICS * First Comp. 0.100 0.836 0.405

AESTHETICS * Control 0.000 . .

iKNOW * Climate Ch. -0.025 -2.986 0.004 **

iKNOW * First Comp. 0.002 0.370 0.712

iKNOW * Control 0.015 2.342 0.021 *

Marginal Model: USEFULNESS

21

Source F Sig.

USABILITY 283.391 < 0.001

AESTHETICS 29.856 < 0.001

TASK-ENV * AESTHETICS 6.961 0.001

TASK-ENV * iKNOW 5.513 0.002

Final Marginal Model

Coefficient Estimates

Higher ratings of USABILITY associated with higher ratings of USEFULNESS.

Higher ratings of AESTHETICS associated with lower ratings of USEFULNESS (but not as much under Climate Change scenario).

Higher iKNOW scores associated with lower ratings of USEFULNESS under Climate Change scenario.

Higher iKNOW scores associated with higher ratings of USEFULNESS under Control scenario.

Marginal Model: GOODNESS

22

Parameter t Sig. TOOL: ipl2 -0.557 0.580 TOOL: RefSeek 2.559 0.013 * TOOL: Awesome Lib. -2.544 0.013 * USEFULNESS 5.464 0.000 ** USABILITY -0.241 0.810 USEFULNESS * Climate Ch. 1.036 0.303 USEFULNESS * First Comp. -3.667 0.001 ** USEFULNESS * Control . . AESTHETICS * Climate Ch. 4.600 < 0.001 ** AESTHETICS * First Comp. 5.699 < 0.001 ** AESTHETICS * Control 0.684 0.497 ENJOYMENT * Climate Ch. -3.071 0.003 ** ENJOYMENT * First Comp. 0.898 0.372 ENJOYMENT * Control 2.850 0.006 ** iKNOW * ipl2 0.015 0.988 iKNOW * RefSeek -2.266 0.027 * iKNOW * Awesome Lib. 2.589 0.012 * USABILITY * ipl2 4.783 < 0.001 ** USABILITY * RefSeek 2.804 0.007 ** USABILITY * Awesome Lib. . . AESTHETICS * ipl2 -4.085 < 0.001 ** AESTHETICS * RefSeek -3.309 0.001 ** AESTHETICS * Awesome Lib. . .

Source F Sig.

TOOL 4.986 0.004

USEFULNESS 38.452 < 0.001

USABILITY 10.059 0.002

TASK-ENV * USEFULNESS 11.397 < 0.001

TASK-ENV * AESTHETICS 12.169 < 0.001

TASK-ENV * ENJOYMENT 6.350 0.001

TOOL * iKNOW 4.463 0.007

TOOL * USABILITY 12.015 < 0.001

TOOL * AESTHETICS 10.339 < 0.001

Final Marginal Model Coefficient Estimates

All variables had some effect on ratings of GOODNESS, but USEFULNESS was the only variable to have a significant relationship across all systems and contexts.

Higher ratings of USEFULNESS were associated with higher ratings of GOODNESS under all three scenarios (but the effect was weaker in the First Computer scenario).

Qualitative Analysis Four themes:

23

Appropriateness to Context

Simplicity and Ease of Use

Pleasurable Interaction

Visual Attractiveness

CODES: - Suitable for Purpose/Goal - Right Functionality - Appropriate Content

CODES: - Easy to Use/Navigate - Speed/Efficiency in Use - Organized/Uncluttered - Streamlined/Simple Design

CODES: - Pleasing to the Eye - Craftsmanship - General Attractiveness

CODES: - Familiarity - “It” Factor - Irritation-Free - Customizability

Usefulness

n=28 77.8%

n=17 47.2%

n=27 75.0%

n=17 47.2%

Usefulness is shaped by context The quantitative analysis showed a significant

effect of contextual factors (the iKnow x Task-Environment interaction) on perceptions of usefulness.

The qualitative analysis revealed that many participants defined a useful website in terms of contextual factors (whether it provides the “right” functions and access to the “right” information).

Conclusion: it is highly likely that the usefulness of

a system depends on the context in which it is used.

24

Usability is linked to usefulness The quantitative analysis showed that higher

ratings of usability were associated with higher ratings of usefulness.

The qualitative analysis revealed that major aspects of usability (efficiency/speed, effectiveness, irritation-free, etc.) were considered aspects of a useful website.

Conclusion: usability seems to be an integral

component of usefulness, but more data is needed (especially at the boundaries).

25

Usefulness may be hedonic? The quantitative analysis showed that websites

seen as more attractive were seen as less useful and that usefulness was not influenced by perception of enjoyment.

The qualitative analysis showed that almost 50% of participants cited some aspect of aesthetics and/or experience in their definitions of a useful website.

Conclusion: the beauty dilemma1 is powerful, and

the experiential aspects of usefulness may be more nuanced than “enjoyment.”

26 Source: 1 Diefenbach & Hassenzahl (2009)

Usefulness matters – a lot The quantitative analysis showed that

usefulness was the only variable with a significant effect on ratings of goodness (regardless of contextual factors). However, other factors (usability, aesthetics,

enjoyment) were also important depending on the tool and/or task-environment.

Conclusion: usefulness is a necessary (but

maybe not sufficient) dimension of a good system.

27

Implications for Research/Practice 1) Evaluators should explicitly address

issues related to usefulness. Probing for usefulness is not uncommon, but it

is infrequent, informal, and without a coherent connection to evaluation goals. (Nørgaard & Hornbæk, 2006)

We encourage evaluators to incorporate questions of usefulness into evaluation plans and purposefully address usefulness during user testing.

28

Implications for Research/Practice 2) Evaluators may consider varying

evaluation contexts. Potential variants: random sampling of tasks,

allowing users to modify test environments, holding tests in various locations, etc.

Doing so would jeopardize the validity of any experiment, but there is plenty of evidence that usability lab experiments are not valid anyway. (Gray & Salzman, 1998; Lindgaard & Chattrichart, 2007; Nørgaard & Hornbæk, 2006; )

29

Limitations & Future Work This study was a laboratory experiment with a

relatively small and tightly controlled group of mostly male tech-savvy undergraduate students and a highly specific type of interface.

We plan to address these limitations in future

work, and encourage other researchers to do so as well.

30

Thank You Craig M. MacDonald cmacdona@pratt.edu http://www.craigmacdonald.com @CraigMMacDonald

31

Acknowledgements: We thank Susan Weidenbeck, Michelle Rogers, Denise Agosto, and

Kasper Hornbæk for their guidance in shaping and directing this research