evaluating user experience
DESCRIPTION
Evaluating User Experience given by Alistair Sutcliffe & Jenny Hart at Manchester NUXTRANSCRIPT
![Page 1: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/1.jpg)
Evaluating User Experience
Alistair Sutcliffe & Jenny Hart
Manchester Business School University of Manchester Manchester M15 6PB,
UK
NUX July 2011 with thanks to Ons AlShamueli & Rabia Khan
![Page 2: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/2.jpg)
Presentation Outline
1. Background to UX research
2. A framework for UI ‘attractiveness’ and user engagement
3. Some experiments on users’ perceptions of design quality
4. Defining User Experience
5. Evaluating User Experience
![Page 3: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/3.jpg)
So What is User Experience (UX) ?
“A person’s perceptions and responses that result from the use or anticipated use of a product, system or service”. ISO 2010
“The effect and affect produced by aesthetic experience, the meaning we attach to the product, and the feelings and emotions produced”
No real agreement on definitions
Essentially beyond the functional- aesthetics, attractiveness – design that excites and holds our attention- user engagement
![Page 4: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/4.jpg)
UX Research- the quants
• Affordances, aesthetics and emotion key factors in design (Norman 2004)
• What is beautiful is usable (Tractinsky 2000, 2004); expressive & classic aesthetics, pleasure,
• Beauty, goodness, pragmatics & hedonics- Attractdiff (Hassenzahl 2004, 2006, 2010)
• Aesthetic perception and interaction (Lingaard 2006, 2009, Hartmann, Sutcliffe & de Angeli 2008) priming effects
![Page 5: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/5.jpg)
UX Research- The Quals (or contextualists)
• Jordon 1998, Pleasure in Products
• Mc Carthy & Wright 2005 Technology As Experience, 2010 User Experience
• Dourish 2004 Where the Action Is Embodied Interaction
• Cockton 2008, Worth maps in UI design
• Hallnäs and Redström,2005, Presence and user experience
• Designers of User Experience- Gaver, Sengers, Forlizzi and many others
![Page 6: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/6.jpg)
UX Research- Viewpoints
1. There are fundamental cognitive constructs by which we perceive User Experience- and hence can evaluate it (Hassenzahl, Tractinsky)
2. User Experience can only be understood in context, each experience is unique. It can only be understood by case study analysis and heuristics (Mc Carthy & Wright + contextualists)
3. User Experience is best understood by design- cultural probes, etc (Gaver and other designers)
4. UX is a cognitive process which can be modelled and measured (Sutcliffe, de Angeli)
![Page 7: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/7.jpg)
The Manchester Attractiveness Framework
Attractiveness
Content / Services
Reputation / Identity
Usability
Aesthetics
Customisability
Attractiveness “Pleasing or appealing to the senses, arousing interest” OED
![Page 8: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/8.jpg)
UX Experiments
• User perceptions of Aesthetics and UI Design qualities - Comparing web sites +/- attractive design features (interactive metaphors, dynamic media) - Sutcliffe & de Angeli INTERACT 05, DIS 06
• Framing effects and Customisation on UX - subject background, task scenario effects, customisation - Hartmann, Sutcliffe & de Angeli CHI 07,08
• Avatars, Immersion and UX - attractive chatterbots & persuasion - comparing UX in immersive v. standard environments (Khan, Sutcliffe & de Angeli, Sutcliffe & AlRayes 2010)
![Page 9: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/9.jpg)
Menu Design style
Both sites- equivalent information, different UI styles
![Page 10: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/10.jpg)
Summary of Differences
Dimensions Engaging style Traditional style Usability - + Aesthetic classic - + Aesthetic expressive
+ -
Information quality
- +
Engagement + - Memory
Overall Preference
Content-based
14 (50%)
Interface-based
14 (50%)
![Page 11: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/11.jpg)
User Judgement: Effect of Customisation
Mobile phone News feed application
2 versions
Good/poor aesthetics
Customised or General content
![Page 12: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/12.jpg)
Results
aesthetics manipulation content-fit manipulation
Measures Aesth++ Standard customised generic
Aesthetic classic + - n/a n/a
Aesthetic
expressive
+ - n/a n/a
Engagement + - n/a n/a
Usability = = = (+) = (-)
Look and Feel + - + -
Customisability = = + -
Content = = + -
Overall
Preference
+ - + -
![Page 13: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/13.jpg)
Summary- Components of Attractiveness
• Attractiveness is a complex mix of factors, but content and services probably more important
• Judgement and choice very dependent on user background and task
• Usability is important but defects will be tolerated if overall experience is positive (halo effect)
• Aesthetics is important but only in context <user background>
• Content and customisation important components in overall preference, but if equal then other criteria come into play
![Page 14: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/14.jpg)
Experiments on Avatars/ Virtual Characters
• Avatars (human like characters) influence preferences
- make information more credible - persuade people more effectively than text/speech alone
• Attractive avatars are more effective than less attractive
- younger, females - similarity to target audience
• Leverages “Computer as Social Actor” effect – we treat representations of people on line like real people (Reeves & Nass 1996)
Khan, Sutcliffe & De Angeli 2010
![Page 15: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/15.jpg)
Attractiveness in Agents
Small differences in appearance make a huge differences in attractiveness and persuasion
![Page 16: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/16.jpg)
Avatars in virtual worlds increase engagement (the Second Life experience)
more interaction improves UX ratings
![Page 17: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/17.jpg)
• UX in the wide
- Experience throughout the Product life cycle - Initial contact (aesthetics) - Use (functionality, content) - Customisation - Support
• UX in the small (interaction, engagement)
-Presence- user as interactor
-Immersion in the interactive world
-Flow and engagement in the interactive world
-Social presence
Components of User Experience
![Page 18: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/18.jpg)
Engagement and Interaction
Interaction with avatars is more engaging if … - They adapt to the user - Use surprise and are occasionally unpredictable
![Page 19: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/19.jpg)
Interaction
User Experience & Engagement
Arousal
Emotion
Media
Pace Flow
Complexity Change
Content Functionality
Presence
Immersion Human Image & Voice
3D Worlds Avatars
increases
promotes
+ve exp induces +ve
![Page 20: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/20.jpg)
Cost Benefit
Need / Specificity
Engagement and Attractiveness- revised framework:
Content services
Reputation Identity
Customis- ability
Aesthetics
Engagement
Usability
Design Quality
Presence Metaphor Interaction
Flow Continuum
All task / context / use dependent
High level impression
Interactive experience
![Page 21: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/21.jpg)
UX over time
Compared 3 web sites – IKEA avatar virtual shop assistant – NIKE animation and customisation – ALDI baseline
Tasks search + interact with features
Same tasks and sites 3 visits separated by 1 week
Hypotheses - character will improve engagement - interaction (customisation) will improve
engagement - effects stronger after more experience
![Page 22: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/22.jpg)
IKEA- virtual character
![Page 23: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/23.jpg)
NIKE customisation
![Page 24: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/24.jpg)
ALDI Ad pop ups
![Page 25: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/25.jpg)
Preferences & ratings
• No significant difference Nike- IKEA
• Preference and rating order the same 1,2 weeks later
• But rating of criteria changed aesthetics, usability earlier, content, brand later
• Avatar (IKEA) not engaging but animation was • Customisation (Nike) not engaging but animation was • ALDI pop ups disliked
![Page 26: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/26.jpg)
Evaluating User Experience (Sutcliffe & Hart 2011)
• Observation (within session) - user activity - body posture, gestures, facial expressions - attention
• Questionnaire (post session) - existing scales- expressive aesthetics + new scales Presence (from VR), Flow-engagement, Media, and immersion - memory recall- salient features, episodes
• Physiological Measures (in session) - heart rate - GSR - pupilometry & eye tracking
![Page 27: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/27.jpg)
Observation
• Activity - system logs or sampling: pace, critical incidents & breakdowns, task v error repair actions
• Attention - gaze on screen/in world v elsewhere - could supplement with eyetracking but expensive
• Non Verbal Communication - post session analysis from video records - rate posture for arousal/excitement - facial expression for emotion
![Page 28: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/28.jpg)
Questionnaires I
General Engagement • Rate your general mood after using the application (positive happy
… negative depressed) • Rate the strength of your feelings/ emotions: Pleasure, Joy,
Surprise, Sadness, Anxiety, Worry, Fear, Frustration, Disgust.
Interaction/Flow • Rate the pace of interaction (too slow, about right, too fast) • How challenging was operating the interface? (too easy, about right,
too difficult)
Media CASA • Did you notice any images of people? (not at all … very much) • How attractive were the images used in the application?
![Page 29: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/29.jpg)
Questionnaires II
Presence • How natural did your interactions with the application feel? • How aware were you of events occurring in the real world around
you?
Immersion • How compelling was your sense of moving inside the interactive
world? • How natural did the interactive world appear to be?
Social Presence • How aware were you of the person you were communicating with? • How well did the application communicate the identity of other
people?
![Page 30: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/30.jpg)
Summary & Conclusions
• User Experience is multi-faceted and will change over time
• It can be measured/systematically evaluated
• UX is context (domain) dependent
• Our judgement of UX suffers from biases and framing effects
• UX components - in session engagement: avatars, virtual worlds and user as
actor- motivate and attract - across session: personalisation, utility, challenge and
adaptation
![Page 31: Evaluating User Experience](https://reader033.vdocuments.mx/reader033/viewer/2022051513/547b3084b479597c098b4d5e/html5/thumbnails/31.jpg)
Thanks for your attention For more information
• Sutcliffe, A.G. (2003). Multimedia and virtual reality: Designing multisensory user interfaces. Mahwah NJ: Lawrence Erlbaum Associates.
• Sutcliffe, A. G. (2009). Designing for user engagement: aesthetic and attractive user interfaces. In Carroll, J.M. (Ed), Synthesis lectures on human centered informatics. San Rafael CA: Morgan Claypool.
• Sutcliffe A.G. & de Angeli A., (2005), Assessing interaction styles in web user interfaces. In Proceedings of Human Computer Interaction INTERACT 05, Eds Costabile M.F and Paterno F., Rome Sept, 2005, Springer Verlag. pp 405-417.
• De Angeli, Sutcliffe A.G. & Hartmann J. (2006) Interaction, usability and aesthetics: What influences users’ preferences? In Proceedings of DIS 2006, Designing Interactive Systems, ACM Press.
• Hartmann J., Sutcliffe A.G. & de Angeli A. (2007), Investigating attractiveness in web user interfaces. in CHI07, Human Factors in Computing Systems, San Jose, CA, ACM Press.
• Hartmann, J., Sutcliffe, A. G., & De Angeli, A. (2008). Framing the user experience: Information biases on website quality judgement. In Proceedings of CHI-08. New York: ACM Press.
• Hartmann, J., Sutcliffe, A. G., & De Angeli, A. (2008). Towards a theory of user judgment of aesthetics and user interface quality. ACM Transactions on Computer-Human Interaction, 15(4), 15-30.