[ixd] week 06. emotional interaction & interfaces
TRANSCRIPT
Lecture 5 & 6
Emotional Interaction & Interfaces
Interaction Design / IID 2017 Spring Class hours : Wednesday 2:00 pm – 5:50 pm Lecture room : International Campus Veritas Hall B308 12th April
Team-up Status
Lecture #5 IID_Interaction Design 2
Name SeeSaw Eevee Lab Abusement Park
P!nk Surprise HI
Lead Rosa Chung So Heum Hwang Jeonghoon Paik Hyunbae Earm Hedgie Choi Jooyeon Shon
Members Susan Kwon Eunwon Cho Phai Tulyathan
Tae Hoon Kim Ha Eun Cho Yena Jang
Joon Young Byun Sabrina Koh Alyosha Kim
Nayeon Kim Ashley Kim Jung Wan Bae
Lucie Benevise Hyo Jeong Park Jisu Lee
Phu Nguyen Jieun Park Anastassiya Kim
Concept Statement
AR/VR based language learning aids for the auditorily impaired
Memory Lane Various SPOT : A map service for pedestrians finding trashcans, smoking areas, and public toilets.
Smartphones for the visually impaired
Smart fridge checking rotten foods
Technical Interest
3D Simulation Place-making by LBS (Maybe)
Amazone Go (?) Collective Intelligence, Crowdsourcing
Tactile interfaces (?)
LG Smart Fridge
Installation Ideas (Devices, SW, Material Composition, Space Setting)
Final Project Concept Statement
• Title
• Members, and each member’s role for the project
• Description of what the project will be about
• The goal of the project
• Who are the users of final outcomes
• Describe the use context
• Outline the technical ideas, and provision
• Add some preliminary sketches of your concept
Lecture #5 IID_Interaction Design 3
Homework
Lecture #5 IID_Interaction Design 4
Team-Up and Outline a Project Concept
Statement Make a Team Blog
[Individual Assignment]
Make Pinterest
Boards for “Interaction” & “Interfaces”
1 2 3
Launch team projects - 4 members - Define your
project goals - Write a Project
Concept Statement
What the blog should contain - Team Name - Team Member links - Post the result of homework1,
“Project Concept Statement”
Read the Main Textbook, chapter 5. “Emotional Interaction,” and chapter 6, “Interfaces” - Collect examples or cases of
“interaction” and “interfaces” of your team project.
- Upload the collections on your pinterest boards, titled “Interaction,” and “Interfaces”
EMOTIONAL INTERACTION
Chapter 5
Lecture #5 IID_Interaction Design 5
Overview
• Emotions and the user experience
• Expressive and annoying interface
– how the ‘appearance’ of an interface can affect users
• Models of emotion
– Ortony et al (2005)
• Automatic emotion recognition and emotional technologies
• Persuasive technologies and behavioral change
– how technologies can be designed to change people’s attitudes and behavior
• Anthropomorphism
– The pros and cons
Lecture #5 IID_Interaction Design 6
Emotions and the user experience
• HCI has traditionally been about designing efficient and effective
systems
• Now more about how to design interactive systems that make people
respond in certain ways
– e.g. to be happy, to be trusting, to learn, to be motivated
• Emotional interaction is concerned with how we feel and react when
interacting with technologies
Lecture #5 IID_Interaction Design 7
Is this form fun to fill in?
• “My goal was to design
Wufoo to feel like
something Fisher-Price
would make.”
• - Kevin Hale, Wufoo director
Lecture #5 IID_Interaction Design 8
Is this form fun to fill in?
Lecture #5 IID_Interaction Design 9
Figure 5.1 Examples of Wufoo's online forms Source: Reproduced with permission from Wufoo.com. http://wufoo.com/examples/#survey.
Emotional interaction
• What makes us happy, sad, annoyed, anxious, frustrated, motivated,
delirious and so on
– translating this into different aspects of the user experience
• Why people become emotionally attached to certain products (e.g. virtual
pets)
• Can social robots help reduce loneliness and improve wellbeing?
• How to change human behavior through the use of emotive feedback
Lecture #5 IID_Interaction Design 10
Activity
• Try to remember the emotions you went through when buying a big
ticket item online (e.g. a fridge, a vacation, a computer)
• How many different emotions did you go through?
Lecture #5 IID_Interaction Design 11
Emotional design model
• Norman, Ortony and Revelle (2004) model of emotion
Lecture #5 IID_Interaction Design 12
Figure 5.2 Ortony, et al (2005) model of emotional design showing three levels: visceral, behavioral, and reflective Source: The illustration and text are from Figure 1.1 of Norman, D. A. (2004). Emotional Design: We love (or hate) everyday things. New York: Basic Books. Reprinted with permission of the author.
Claims from model
• Our emotional state changes how we think
– when frightened or angry we focus narrowly and body responds by
tensing muscles and sweating
• more likely to be less tolerant
– when happy we are less focused and the body relaxes
• more likely to overlook minor problems and be more creative
Lecture #5 IID_Interaction Design 13
Activity
• Do you feel more creative when you are in a happy mood? Do you get
less work done when you are feeling stressed?
Lecture #5 IID_Interaction Design 14
Video of Kia Höök talking about affective interaction in a series of videos that consider emotion in terms of how it is constructed in interaction with technology and people is available at www.interaction-design.org/encyclopedia/affective_computing.html
Expressive interfaces
• Provide reassuring feedback that can be both informative and fun
• But can also be intrusive, causing people to get annoyed and even
angry
• Color, icons, sounds, graphical elements and animations are used to
make the ‘look and feel’ of an interface appealing
– conveys an emotional state
• In turn this can affect the usability of an interface
– people are prepared to put up with certain aspects of an interface (e.g.
slow download rate) if the end result is appealing and aesthetic
Lecture #5 IID_Interaction Design 15
Friendly interfaces
• Microsoft pioneered friendly interfaces for technophobes - ‘At home
with Bob’ software
• 3D metaphors based on familiar places (e.g. living rooms)
• Agents in the guise of pets (e.g. bunny, dog) were included to talk to
the user
– Make users feel more at ease and comfortable
Lecture #5 IID_Interaction Design 16
Bob
Lecture #5 IID_Interaction Design 17
Figure 5.6 ‘At home with Bob’ software developed for Windows 95. Although now defunct, it has been resurrected affectionately to run on a Virtual PC platform Source: Microsoft product screenshot reproduced with permission from Microsoft Corporation.
Clippy
• Why was Clippy disliked by so
many?
• Was it annoying, distracting,
patronising or other?
• What sort of user liked Clippy?
Lecture #5 IID_Interaction Design 18
Figure 5.7 Microsoft's agent Clippy Source: Microsoft product screenshot reproduced with permission from Microsoft Corporation.
Frustrating interfaces
• Many causes:
– When an application doesn’t work properly or crashes
– When a system doesn’t do what the user wants it to do
– When a user’s expectations are not met
– When a system does not provide sufficient information to enable the user to
know what to do
– When error messages pop up that are vague, obtuse or condemning
– When the appearance of an interface is garish, noisy, gimmicky or patronizing
– When a system requires users to carry out too many steps to perform a task,
only to discover a mistake was made earlier and they need to start all over
again
Lecture #5 IID_Interaction Design 19
Activity 5.3
• Most of us are familiar with the ‘404 error’ message that pops up now
and again when our computer doesn't upload the web page we're
trying to view. But what does it mean and why the number 404? How
does it make you feel when you see it and what do you do next? Is
there a better way of letting users know when they or the computer
have made an error?
Lecture #5 IID_Interaction Design 20
Activity 5.3
• Comment
– The main reasons for this error message appearing are:
• (i) you have typed in the URL incorrectly,
• (ii) you have followed an old link that is now broken, or
• (iii) you have tried to access a page that is blocked by an ISP.
– The number 404 comes from the HTML language.
• The first 4 indicates a client error. The server is telling you that you've done something
wrong, such as misspelt the URL or requested a page that no longer exists.
• The middle 0 refers to a general syntax error, such as a spelling mistake.
• The last 4 indicates the specific nature of the error.
– For the user, however, it is an arbitrary number. It might even suggest that there are 400
other errors they could make!
Lecture #5 IID_Interaction Design 21
Error messages
– “The application Word Wonder has unexpectedly quit due to a type 2 error.”
• Why not instead:
– “the application has expectedly quit due to poor coding in the operating system”
• Shneiderman’s guidelines for error messages include:
– avoid using terms like FATAL, INVALID, BAD
– Audio warnings
– Avoid UPPERCASE and long code numbers
– Messages should be precise rather than vague
– Provide context-sensitive help
Lecture #5 IID_Interaction Design 22
Website error messages
Lecture #5 IID_Interaction Design 23
Figure 5.8 An error message that appears if a user types in his or her personal details for accessing the protected website incorrectly
Should computers say they’re sorry?
• Reeves and Naas (1996) argue that computers should be made to
apologize
• Should emulate human etiquette
• Would users be as forgiving of computers saying sorry as people are
of each other when saying sorry?
• How sincere would they think the computer was being? For example,
after a system crash:
– “I’m really sorry I crashed. I’ll try not to do it again”
• How else should computers communicate with users?
Lecture #5 IID_Interaction Design 24
Detecting emotions and emotional technology
• Sensing technologies used to measure GSR, facial expressions,
gestures, body movement
• Aim is to predict user’s emotions and aspects of their behavior –
• E.g. what is someone most likely to buy online when feeling sad, bored
or happy
Lecture #5 IID_Interaction Design 25
Facial Coding
• Measures a user’s emotions as they interact with a computer or tablet
• Analyses images captured by a webcam of their face
• Uses this to gauge how engaged the user is when looking at movies,
online shopping sites and ads
• 6 core expressions - sadness, happiness, disgust, fear, surprise and
anger
Lecture #5 IID_Interaction Design 26
Six fundamental emotions sadness, happiness, disgust, fear, surprise, and anger
Lecture #5 IID_Interaction Design 27
Figure 5.9 A screen shot showing facial coding from Affdex software Source: Courtesy of Affectiva, Inc.
How to use the emotional data?
• If user screws up their face when an ad pops up -> feel disgust
• If start smiling -> they are feeling happy
• Website can adapt its ad, movie storyline or content to match user’s
emotional state
• Eye-tracking, finger pulse, speech and words/phrases also analysed
when tweeting or posting to Facebook
Lecture #5 IID_Interaction Design 28
Indirect emotion detection
• Beginning to be used more to infer or predict someone’s behavior
• For example, determining a person’s suitability for a job, or how they
will vote at an election
• Do you think it is creepy that technology can read your emotions from
your facial expressions or from your tweets?
Lecture #5 IID_Interaction Design 29
Moon Phrases(de Choudhury et al, 2013)
• An app that has been developed to help people
reflect upon their emotional well-being
• The aim is to help them cope better with stress
and anxiety by recognizing the triggers that cause
them to occur.
• By looking back at their history of moons, users
can start to identify patterns that could help them
understand more about themselves and what
might be causing their mood swings.
Lecture #5 IID_Interaction Design 30
Persuasive technologies and behavioral change
• Interacive computing systems deliberately designed to change
people’s attitudes and behaviors (Fogg, 2003)
• A diversity of techniques now used to change what they do or think
• Pop-up ads, warning messages, reminders, prompts, personalized
messages, recommendations, Amazon 1-click
• Commonly referred to as nudging
Lecture #5 IID_Interaction Design 31
Nintendo’s Pocket Pikachu
• Changing bad habits and
improving well being
– Designed to motivate children to
be more physically active on a
regular basis
– Owner of the digital pet that
‘lives’ in the device is required
to walk, run, or jump
– If owner does not exercise the
virtual pet becomes angry and
refuses to play anymore
Lecture #5 IID_Interaction Design 32
How effective?
• Is the use of novel forms of interactive technologies (e.g., the
combination of sensors and dynamically updated information) that
monitor, nag, or send personalized messages intermittently to a person
more effective at changing a person’s behavior than non-interactive
methods, such as the placement of warning signs, labels, or ads in
prominent positions?
Lecture #5 IID_Interaction Design 33
Activity 5.4
• Watch the two videos:
Lecture #5 IID_Interaction Design 34
Activity 5.4
• Watch the two videos:
Lecture #5 IID_Interaction Design 35
Activity 5.4
• Comment
– Volkswagen sponsored an open competition, called the fun theory, asking
people to transform mundane artifacts into novel enjoyable user
experiences in an attempt to change people's behavior for the better
(www.thefuntheory.com). The idea was to encourage a desired behavior
through making it more fun. The Piano Staircase and the Outdoor Bin are
the most well known examples; the stairs sounded like piano keys being
played when walked on while the bin sounded like a well echoing when
something was thrown into it
Lecture #5 IID_Interaction Design 36
Activity 5.5
• What do you think might be problematic when using this kind of dieting
device in a restaurant?
Lecture #5 IID_Interaction Design 37
Figure 5.10 A friend using the HAPIfork in a restaurant when eating a cake
Activity 5.5
• Comment
– The photo shows a friend using the device to eat a cake in a restaurant.
He first commented on the fork being awkward to hold. He also felt self-
conscious using a brightly coloured fork in a public setting that was very
different from what everyone else was using to eat their food. The waiters
also gave him funny looks when using it. He said it might be fun to try
once but not to use for every meal and that he could not see himself or
anyone else he knew using it in public.
Lecture #5 IID_Interaction Design 38
Which is most effective?
Lecture #5 IID_Interaction Design 39
Figure 5.11 Stairs versus elevators: static sign versus ambient persuasive displays. Which is most persuasive?
Activity 5.6
• The photo on the left in Figure 5.11 is of a banner placed in downtown LA, in an attempt
to encourage the general public to take the stairs instead of the elevator, asking people
to climb stairs on a certain day.
• The two photos on the right are of ambient displays (see also Chapter 2) designed to do
the same thing but using more subtle and interactive methods:
– (i) lights that twinkled when people approach them, intended to lure them to take the stairs and
– (ii) clouds of different colored spheres that move up and down depending on how many people
have taken the stairs or the elevator for a given period of time (gray represents elevator).
– The higher the orange cloud is relative to the gray one, the more people are taking the stairs
than the elevator (Rogers, Hazlewood, Marshall et al, 2010).
• Which representation do you think is the most effective?
Lecture #5 IID_Interaction Design 40
Activity 5.6
• Comment
– The banner on the left uses a striking slogan that metaphorically points a finger at you. Some people
might find this amusing or a challenge. Others, however, are likely to find it offensive and even insulting
and ignore it. The extent to which it will change behavior is debatable. In contrast, the ambient
displays shown in the other images are meant to be much more subtle and aesthetic.
– They are assumed to work by raising people's awareness of a behavior that they normally overlook or
try not to think about and nudge them at the point of decision making (Thaler and Sunstein, 2008). An
in-the-wild study conducted when they were embedded in a building showed a significant increase in
stair taking.
– Interestingly, the majority of people, when asked, were not aware of changing their behavior in
response to noticing them. This suggests that ambient displays can influence at an unconscious level,
and they can be used to encourage people to adhere to certain kinds of desired behaviors, where
there is a choice, such as washing hands or not, eating healthy or unhealthy snacks, taking one route
or another through a building, switching off the lights or leaving them on.
Lecture #5 IID_Interaction Design 41
Tracking devices
• Mobile apps designed to help people monitor and change their
behaviour (e.g. fitness, sleeping, weight)
• Can compare with online leader boards and charts, to show how they
have done in relation to their peers and friends
• Also apps that encourage reflection that in turn increase well-being
and happiness
Lecture #5 IID_Interaction Design 42
Energy reduction
Lecture #5 IID_Interaction Design 43
Figure 5.12 (a) The Power Aware Cord consists of an electrical power strip in which the cord is designed to visualize the energy rather than hiding it. Increase and decrease in use is conveyed through showing glowing pulses, flow, and intensity of light. (b) The Waatson (now a commercial product available in many countries) measures in watts or cost how much electricity someone is using in their home at any moment. This is conveyed in LEDs on the top side. On the underside are colored lights: when they glow blue it shows you are using less than normal; when it changes to purple it indicates that your usage is average: and when it is red it indicates you are using more than normal Source: (a) Photo taken from the Interactive Institute's research program “Static!” and reproduced with permission. (b) Reproduced with permission from DIY Kyoto Ltd. www.diykyoto.com.
The Tidy Street project
• large-scale visualization of the street’s electricity usage
– stenciled display on the road surface using chalk
– provided realtime feedback that all could see change each day
– reduced electricity consumption by 15%
Lecture #5 IID_Interaction Design 44
Figure 5.13 Looking down at the Tidy Street public electricity graph from a bedroom window
Phishing and trust
• Web used to deceive people into parting with personal details
– e.g. Paypal, eBay and won the lottery letters
• Allows Internet fraudsters to access their bank accounts and draw money
from them
• Many vulnerable people fall for it
• The art of deception is centuries old but internet allows ever more
ingenious ways to trick people
Lecture #5 IID_Interaction Design 45
Anthropomorphism
• Attributing human-like qualities to inanimate objects (e.g. cars,
computers)
• Well known phenomenon in advertising
– Dancing butter, drinks, breakfast cereals
• Much exploited in human-computer interaction
– Make user experience more enjoyable, more motivating, make people feel
at ease, reduce anxiety
Lecture #5 IID_Interaction Design 46
Which do you prefer?
1. As a welcome message
Lecture #5 IID_Interaction Design 47
Hello Chris!
Nice to see you again. Welcome back. Now what were we doing last time?
Oh yes, exercise 5.
Let’s start again.
Which do you prefer?
1. As a welcome message
Lecture #5 IID_Interaction Design 48
User 24
commence exercise 5.
Which do you prefer?
2. Feedback when get something wrong
Lecture #5 IID_Interaction Design 49
Now Chris, that’s not right. You can do better than that. Try again.
Which do you prefer?
2. Feedback when get something wrong
Lecture #5 IID_Interaction Design 50
Incorrect. Try again.
Which do you prefer?
2. Feedback when get something wrong
Lecture #5 IID_Interaction Design 51
Incorrect. Try again.
Is there a difference as to what you
prefer depending on type of message?
Why?
Evidence to support anthropomorphism
• Reeves and Naas (1996) found that computers that flatter and praise
users in education software programs >> positive impact on them
• “Your question makes an important and useful distinction. Great job!”
• Students were more willing to continue with exercises with this kind of
feedback
Lecture #5 IID_Interaction Design 52
Criticism of anthropomorphism
• Deceptive, make people feel anxious, inferior or stupid
• People tend not to like screen characters that wave their fingers at the
user and say:
– Now Chris, that’s not right. You can do better than that. Try again.”
• Many prefer the more impersonal:
– “Incorrect. Try again.”
• Studies have shown that personalized feedback is considered to be
less honest and makes users feel less responsible for their actions
(e.g. Quintanar, 1982)
Lecture #5 IID_Interaction Design 53
Virtual characters
• Appearing on our screens in the form of:
– Sales agents, characters in videogames, learning companions, wizards,
pets, newsreaders
• Provides a persona that is welcoming, has personality and makes user
feel involved with them
Lecture #5 IID_Interaction Design 54
Disadvantages
• Can lead people into false sense of belief, enticing them to confide
personal secrets with chatterbots
• Annoying and frustrating
– e.g. Clippy
• May not be trustworthy
– virtual shop assistants?
Lecture #5 IID_Interaction Design 55
Virtual agents
• What do the virtual agents do?
• Do they elicit an emotional response in you?
• Do you trust them?
• What is the style of interaction?
• What facial expression do they have?
• Are they believable, pushy, helpful?
• Would it be different if they were male? If so, how?
Lecture #5 IID_Interaction Design 56
What makes a virtual agent believable?
• Believability refers to the extent to which users come to believe an
agent’s intentions and personality
• Appearance is very important
– Are simple cartoon-like characters or more realistic characters,
resembling the human form more believable?
• Behaviour is very important
– How an agent moves, gestures and refers to objects on the screen
– Exaggeration of facial expressions and gestures to show underlying
emotions (c.f. animation industry)
Lecture #5 IID_Interaction Design 57
Activity 5.7
• Early robot pets, such as Sony's AIBO and QRIO, were made of metal
and looked like robots. In contrast, a more recent trend has been to
make robot pets feel and look more like real pets by covering them in
fur (e.g. squirrels, cats, rabbits) and making them behave in more cute
pet-like ways.
• Two contrasting examples are presented in Figure 5.14. Which do you
prefer and why?
Lecture #5 IID_Interaction Design 58
Robot-like or cuddly?
• Which do you prefer and why?
Lecture #5 IID_Interaction Design 59
Figure 5.14 Two kinds of robot pets Source: (a) Courtesy of Sony Corporation, (b) Reproduced with permission of Steve Yohanan. Photo: Martin Dee.
The Haptic Creature
Lecture #5 IID_Interaction Design 61
Activity 5.7
• Comment
– Most people like stroking pets and cuddly toys and so will likely want to stroke pet robots. However,
because they look and feel like the real thing some people may find them a bit creepy. A motivation for
making robot pets cuddly is to enhance the emotional experience people have through using their
sense of touch.
– For example, the Haptic Creature on the right is a robot that mimics a pet that might sit in your lap,
such as a cat or a rabbit (Yohanan and MacLean, 2008). It is made up of a body, head, and two ears, as
well as mechanisms that simulate breathing, a vibrating purr, and the warmth of a living creature. The
robot ‘detects’ the way it is touched by means of an array of (roughly 60) touch sensors laid out across
its whole body and an accelerometer.
– When the Haptic Creature is stroked, it responds accordingly, using the ears, breathing, and purring to
communicate its emotional state through touch. On the other hand, the sensors are also used by the
robot to detect the human's emotional state through touch. Note how the robot has no eyes, nose, or
mouth. Facial expressions are the most common way humans communicate emotional states. Since
the Haptic Creature communicates and senses emotional states solely through touch, the face was
deliberately left off to prevent people trying to ‘read’ emotion from it.
Lecture #5 IID_Interaction Design 62
Implications
• Should we create products that adapt according to people’s different
emotional states?
– When people are feeling angry should an interface be more attentive and
informative than when they are happy?
• Is Norman right?
– designers “can get away with more” for products intended to be used
during leisure time than those designed for serious tasks
Lecture #5 IID_Interaction Design 63
Summary
• Emotional aspects of interaction design concerned with how to facilitate certain
states (e.g. pleasure) or avoid reactions (e.g. frustration)
• Well-designed interfaces can elicit good feelings in people
• Aesthetically pleasing interfaces can be a pleasure to use
• Expressive interfaces can provide reassuring feedback to users
• Badly designed interfaces make people frustrated, annoyed, or angry
• Emotional technologies can be designed to persuade people to change their
behaviors or attitudes
• Anthropomorphism is the attribution of human qualities to objects
• Virtual agents and robot pets have been developed to make people feel motivated,
reassured, and in a good mood
Lecture #5 IID_Interaction Design 64
INTERFACES
Chapter 6
Lecture #6 IID_Interaction Design 65
Overview
• Interface types
– highlight the main design and research issues for each of the different
interfaces
• Consider which interface is best for a given application or activity
Lecture #6 IID_Interaction Design 66
Interface Types
Lecture #6 IID_Interaction Design 67
Interface type See also
1. Command-based 2. WIMP and GUI 3. Multimedia 4. Virtual reality 5. Information visualization and dashboards 6. Web 7. Consumer electronics and appliances 8. Mobile 9. Speech 10. Pen 11. Touch 12. Air-based gesture 13. Haptic 14. Multimodal 15. Shareable 16. Tangible 17. Augmented and mixed reality 18. Wearable 19. Robots and drones 20. Brain–computer interaction (BCI)
WIMP and web Augmented and mixed reality Multimedia Mobile and multimedia Mobile Augmented and mixed reality Shareable, touch Shareable, air-based gesture Tangible Multimodal Speech, pen, touch, gesture, and haptic Touch Virtual reality
1. Command-based
• Commands such as abbreviations (e.g. ls) typed in at the prompt to
which the system responds (e.g. listing current files)
• Some are hard wired at keyboard, others can be assigned to keys
• Efficient, precise, and fast
• Large overhead to learning set of commands
Lecture #6 IID_Interaction Design 68
Second Life command-based interface for visually impaired users
Lecture #6 IID_Interaction Design 69
Figure 6.1 Second Life command-based interface for visually impaired users Source: Reproduced with permission from http://www.eelke.com/images/textsl.jpg.
Research and design issues
• Form, name types and structure are key research questions
• Consistency is most important design principle
– e.g. always use first letter of command
• Command interfaces popular for web scripting
Lecture #6 IID_Interaction Design 70
2. WIMP and GUI
• Xerox Star first WIMP → rise to GUIs
• Windows
– could be scrolled, stretched, overlapped, opened, closed, and moved around the screen
using the mouse
• Icons
– represented applications, objects, commands, and tools that were opened when clicked
on
• Menus
– offering lists of options that could be scrolled through and selected
• Pointing device
– a mouse controlling the cursor as a point of entry to the windows, menus, and icons on
the screen
Lecture #6 IID_Interaction Design 71
GUIs
• Same basic building blocks as WIMPs but more varied
– Color, 3D, sound, animation,
– Many types of menus, icons, windows
• New graphical elements, e.g.
– toolbars, docks, rollovers
• Challenge now is to design GUIs that are best suited for tablet,
smartphone and smartwatch interfaces
Lecture #6 IID_Interaction Design 72
Windows
• Windows were invented to overcome physical constraints of a
computer display
– enable more information to be viewed and tasks to be performed
• Scroll bars within windows also enable more information to be viewed
• Multiple windows can make it difficult to find desired one
– listing, iconising, shrinking are techniques that help
Lecture #6 IID_Interaction Design 73
Lecture #6 IID_Interaction Design 74
Figure 6.2 The boxy look of the fi rst generation of GUIs. The window presents several check boxes, notes boxes, and options as square buttons Source: Mullet, Kevin; Sano, Darrell, Designing Visual Interfaces: Communication Oriented Techniques, 1st, © 1995. Reproduced by permission of Pearson Education, Inc., Upper Saddle River, New Jersey.
Apple’s shrinking windows
Lecture #6 IID_Interaction Design 75
Safari panorama window view
Lecture #6 IID_Interaction Design 76
Figure 6.3 A window management technique provided in Safari: pressing the 4 × 3 icon in the top left corner of the bookmarks bar displays the 12 top sites visited, by shrinking them and placing them side by side. This enables the user to see them all at a glance and be able to rapidly switch between them
Selecting a country from a scrolling window
Lecture #6 IID_Interaction Design 77
Figure 6.4 A scrolling menu Source: Screenshot of Camino browser, ©The Camino Project.
Is this method any better?
Lecture #6 IID_Interaction Design 78
Figure 6.6 Two different menu layouts Source: (a) http://www.jamieoliver.com/italian/menu (b) http://www.tonysasianfusion.com/japanesemenu.html.
Research and design issues
• Window management
– enables users to move fluidly between different windows (and monitors)
• How to switch attention between windows without getting distracted
• Design principles of spacing, grouping, and simplicity should be used
Lecture #6 IID_Interaction Design 79
Menus
• A number of menu interface styles
– flat lists, drop-down, pop-up, contextual, and expanding ones, e.g.,
scrolling and cascading
• Flat menus
– good at displaying a small number of options at the same time and where
the size of the display is small, e.g. iPods
– but have to nest the lists of options within each other, requiring several
steps to get to the list with the desired option
– moving through previous screens can be tedious
Lecture #6 IID_Interaction Design 80
Expanding menus
• Enables more options to be shown on a single screen than is possible with
a single flat menu
• More flexible navigation, allowing for selection of options to be done in
the same window
• Most popular are cascading ones
– primary, secondary and even tertiary menus
– downside is that they require precise mouse control
– can result in overshooting or selecting wrong options
Lecture #6 IID_Interaction Design 81
Cascading menu
Lecture #6 IID_Interaction Design 82
Figure 6.7 A cascading menu
Contextual menus
• Provide access to often-used commands that make sense in the context
of a current task
• Appear when the user presses the Control key while clicking on an
interface element
– e.g., clicking on a photo in a website together with holding down the Control
key results in options ‘open it in a new window,’ ‘save it,’ or ‘copy it’
• Helps overcome some of the navigation problems associated with
cascading menus
Lecture #6 IID_Interaction Design 83
Windows Jump List Menu
Lecture #6 IID_Interaction Design 84
Figure 6.8 Windows jump list Source: http://windows.microsoft.com/en-US/windows7/products/features/jump-lists.
Research and design issues
• What are best names/labels/phrases to use?
• Placement in list is critical
– Quit and save need to be far apart
• Choice of menu to use determined by application and type of system
– flat menus are best for displaying a small number of options at one time
– expanding menus are good for showing a large number of options
Lecture #6 IID_Interaction Design 85
Icon design
• Icons are assumed to be easier to learn and remember than
commands
• Can be designed to be compact and variably positioned on a screen
• Now pervasive in every interface
– e.g. represent desktop objects, tools (e.g. paintbrush), applications (e.g.
web browser), and operations (e.g. cut, paste, next, accept, change)
Lecture #6 IID_Interaction Design 86
Icons
• Since the Xerox Star days icons have changed in their look and feel:
– black and white → color, shadowing, photorealistic images, 3D rendering,
and animation
• Many designed to be very detailed and animated making them both
visually attractive and informative
• GUIs now highly inviting, emotionally appealing, and feel alive
Lecture #6 IID_Interaction Design 87
Icon forms
• The mapping between the representation and underlying referent can be:
– similar (e.g., a picture of a file to represent the object file)
– analogical (e.g., a picture of a pair of scissors to represent ‘cut)
– arbitrary (e.g., the use of an X to represent ‘delete’)
• Most effective icons are similar ones
• Many operations are actions making it more difficult to represent them
– use a combination of objects and symbols that capture the salient part of an
action
Lecture #6 IID_Interaction Design 88
Early icons
Lecture #6 IID_Interaction Design 89
Figure 6.9 Poor icon set from the early 1990s. What do you think they mean and why are they so bad? Source:K. Mullet and D. Sano: “Designing Visual Interfaces” Pearson 1995, reproduced with permission of Pearson Education.
Early icons
Lecture #6 IID_Interaction Design 90
Figure 6.10 Early and more recent Mac icon designs for the TextEdit application
Newer icons
Lecture #6 IID_Interaction Design 91
Figure 6.11 Contrasting genres of Aqua icons used for the Mac. The top row of icons have been designed for user applications and the bottom row for utility applications
Simple flat 2D icons
Lecture #6 IID_Interaction Design 92
Figure 6.12 Flat 2D icons designed for smartphone apps
Activity
• Sketch simple icons to represent the following operations to appear
on a digital camera screen:
– Turn image 90 degrees sideways
– Auto-enhance the image
– Fix red-eye
– Crop the image
• Show them to someone else and see if they can understand what
each represents
Lecture #6 IID_Interaction Design 93
Basic edit icons on iPhone
• Which is which?
• Are they easy to understand
• Are they distinguishable?
• What representation forms
are used?
• How do yours compare?
Lecture #6 IID_Interaction Design 94
Research and design issues
• There is a wealth of resources now so do not have to draw or invent
new icons from scratch
– guidelines, style guides, icon builders, libraries
• Text labels can be used alongside icons to help identification for small
icon sets
• For large icon sets (e.g. photo editing or word processing) use
rollovers
Lecture #6 IID_Interaction Design 95
3. Multimedia
• Combines different media within a single interface with various forms
of interactivity
– graphics, text, video, sound, and animations
• Users click on links in an image or text
→ another part of the program
→ an animation or a video clip is played
→ can return to where they were or move on to another place
Lecture #6 IID_Interaction Design 96
BioBlast Multimedia Learning Environment
Lecture #6 IID_Interaction Design 97
Figure 6.14 Screen dump from the multimedia environment BioBLAST Source: Screenshot from BioBlast, ©Wheeling Jesuit University.
Pros and cons
• Facilitates rapid access to multiple representations of information
• Can provide better ways of presenting information than can any media alone
• Can enable easier learning, better understanding, more engagement, and more
pleasure
• Can encourage users to explore different parts of a game or story
• Tendency to play video clips and animations, while skimming through
accompanying text or diagrams
Lecture #6 IID_Interaction Design 98
Research and design issues
• How to design multimedia to help users explore, keep track of, and
integrate the multiple representations
– provide hands-on interactivities and simulations that the user has to
complete to solve a task
– Use ‘dynalinking,’ where information depicted in one window explicitly
changes in relation to what happens in another (Scaife and Rogers, 1996).
• Several guidelines that recommend how to combine multiple media for
different kinds of task
Lecture #6 IID_Interaction Design 99
4. Virtual reality
• Computer-generated graphical simulations providing:
– “the illusion of participation in a synthetic environment rather than
external observation of such an environment” (Gigante, 1993)
• Provide new kinds of experience, enabling users to interact with
objects and navigate in 3D space
• Create highly engaging user experiences
Lecture #6 IID_Interaction Design 100
Pros and cons
• Can have a higher level of fidelity with objects they represent compared to
multimedia
• Induces a sense of presence where someone is totally engrossed by the
experience
– “a state of consciousness, the (psychological) sense of being in the virtual environment”
(Slater and Wilbur, 1999)
• Provides different viewpoints: 1st and 3rd person
• Head-mounted displays are uncomfortable to wear, and can cause motion sickness
and disorientation
Lecture #6 IID_Interaction Design 101
Research and design issues
• Much research on how to design safe and realistic VRs to facilitate
training
– e.g. flying simulators
– help people overcome phobias (e.g. spiders, talking in public)
• Design issues
– how best to navigate through them (e.g. first versus third person)
– how to control interactions and movements (e.g. use of head and body
movements)
– how best to interact with information (e.g. use of keypads, pointing, joystick
buttons);
– level of realism to aim for to engender a sense of presence
Lecture #6 IID_Interaction Design 102
Realism vs. Abstraction?
Lecture #6 IID_Interaction Design 103
Figure 6.15 Magic Cap's 3D desktop interface Source: Reprinted by permission of General Magic Inc.
Which is the most engaging game of Snake?
Lecture #6 IID_Interaction Design 104
Figure 6.16 Two screenshots from the game Snake – the one on the left is played on a PC and the one on the right on a cell phone. In both games, the goal is to move the snake (the blue thing and the black squares, respectively) towards targets that pop up on the screen (e.g. the bridge, the star) and to avoid obstacles (e.g. a flower, the end of the snake's tail). When a player successfully moves his snake head over or under a target, the snake increases its length by one blob or block. The longer the snake gets, the harder it is to avoid obstacles. If the snake hits an obstacle, the game is over. On the PC version there are lots of extra features that make the game more complicated, including more obstacles and ways of moving. The cell phone version has a simple 2D bird's eye representation, whereas the PC version adopts a 3D third-person avatar perspective
5. Information visualization and dashboards
• Computer-generated interactive graphics of complex data
• Amplify human cognition, enabling users to see patterns, trends, and
anomalies in the visualization (Card et al, 1999)
• Aim is to enhance discovery, decision-making, and explanation of
phenomena
• Techniques include:
– 3D interactive maps that can be zoomed in and out of and which present data
via webs, trees, clusters, scatterplot diagrams, and interconnected nodes
Lecture #6 IID_Interaction Design 105
Dashboards
• Show screenshots of data updated over periods of time - to be read at a
glance
• Usually not interactive - slices of data that depict current state of a system
or process
• Need to provide digestible and legible information for users
– design its spatial layout so intuitive to read when first looking at it
– should also direct a user’s attention to anomalies or unexpected deviations
Lecture #6 IID_Interaction Design 106
Dashboards
Lecture #6 IID_Interaction Design 107
Figure 6.17 An info visualization, using flat colored blocks Source: Reproduced with permission from Fekete, J.D., Plaisant, C., Interactive Information Visualization of a Million Items, Proc. IEEE Symposium on Information Visualization (2002), 117–124. www.cs.umd.edu/hcil/millionvis.
Which dashboard is best?
Lecture #6 IID_Interaction Design 108
Figure 6.18 Screenshots from two dashboards: (a) British Airways frequent flier club that shows how much a member has flown since joining them, and (b) London City that provides various information feeds. Which is the easier to read and most informative?
(a)
Which dashboard is best?
Lecture #6 IID_Interaction Design 109
Figure 6.18 Screenshots from two dashboards: (a) British Airways frequent flier club that shows how much a member has flown since joining them, and (b) London City that provides various information feeds. Which is the easier to read and most informative?
(b)
Research and design issues
• Whether to use animation and/or interactivity
• What form of coding to use, e.g. color or text labels
• Whether to use a 2D or 3D representational format
• What forms of navigation, e.g. zooming or panning,
• What kinds and how much additional information to provide, e.g.
rollovers or tables of text
• What navigational metaphor to use
Lecture #6 IID_Interaction Design 110
6. Web
• Early websites were largely text-based, providing hyperlinks
• Concern was with how best to structure information to enable users to navigate
and access it easily and quickly
• Nowadays, more emphasis on making pages distinctive, striking, and pleasurable
• Need to think of how to design information for multi-platforms - keyboard or touch?
– e.g. smartphones, tablets, PCs
Lecture #6 IID_Interaction Design 111
Usability versus attractive?
• Vanilla or multi-flavor design?
– Ease of finding something versus aesthetic and enjoyable experience
• Web designers are:
– “thinking great literature”
• Users read the web like a:
– “billboard going by at 60 miles an hour” (Krug, 2000)
• Need to determine how to brand a web page to catch and keep ‘eyeballs’
Lecture #6 IID_Interaction Design 112
In your face ads
• Web advertising is often intrusive and pervasive
• Flashing, aggressive, persistent, annoying
• Often need to be ‘actioned’ to get rid of
• What is the alternative?
Lecture #6 IID_Interaction Design 113
Research and design issues
• Need to consider how best to design, present, and structure
information and system behavior
• But also content and navigation are central
• Veen’s (2001) design principles
1) Where am I?
2) Where can I go?
3) What’s here?
Lecture #6 IID_Interaction Design 114
Activity
• Look at the Nike.com website
• What kind of website is it?
• How does it contravene the design principles outlined by Veen?
• Does it matter?
• What kind of user experience is it providing for?
• What was your experience of engaging with it?
Lecture #6 IID_Interaction Design 115
7. Consumer electronics and appliances
• Everyday devices in home, public place, or car
– e.g. washing machines, remotes, photocopiers, printers and navigation systems)
• And personal devices
– e.g. MP3 player, digital clock and digital camera
• Used for short periods
– e.g. putting the washing on, watching a program, buying a ticket, changing the time,
taking a snapshot
• Need to be usable with minimal, if any, learning
Lecture #6 IID_Interaction Design 116
Research and design issues
• Need to design as transient interfaces with short interactions
• Simple interfaces
• Consider trade-off between soft and hard controls
– e.g. buttons or keys, dials or scrolling
Lecture #6 IID_Interaction Design 117
8. Mobile
• Handheld devices intended to be used while on the move
• Have become pervasive, increasingly used in all aspects of everyday and
working life
• Apps running on mobiles have greatly expanded, e.g.
– used in restaurants to take orders
– car rentals to check in car returns
– supermarkets for checking stock
– in the streets for multi-user gaming
– in education to support life-long learning
Lecture #6 IID_Interaction Design 118
The advent of the iPhone app
• A whole new user experience that was designed primarily for people
to enjoy
– many apps not designed for any need, want or use but purely for idle
moments to have some fun
– e.g. iBeer developed by magician Steve Sheraton
– ingenious use of the accelerometer that is inside the phone
Lecture #6 IID_Interaction Design 119
iBeer app
Lecture #6 IID_Interaction Design 120
Figure 6.20 The iBeer smartphone app Source: iBeer™ Photo ©2010 HOTTRIX® Reproduced with permission.
QR codes and cell phones
Lecture #6 IID_Interaction Design 121
Figure 6.21 QR code appearing on a magazine page
Mobile challenges
• Smaller screens, small number of physical keys and restricted number of controls
• Innovative physical designs including:
– roller wheels, rocker dials, up/down ‘lips’ on the face of phones, 2-way and 4-way
directional keypads, softkeys, silk-screened buttons
• Usability and preference varies
– depends on the dexterity and commitment of the user
• Smartphones overcome mobile physical constraints through using multi-touch
displays
Lecture #6 IID_Interaction Design 122
Research and design issues
• Mobile interfaces can be tricky and cumbersome to use for those with
poor manual dexterity or ‘fat’ fingers
• Key concern is hit area
– area on the phone display that the user touches to make something
happen, such as a key, an icon, a button or an app
– space needs to be big enough for fat fingers to accurately press
– if too small the user may accidentally press the wrong key
Lecture #6 IID_Interaction Design 123
9. Speech
• Where a person talks with a system that has a spoken language
application, e.g. timetable, travel planner
• Used most for inquiring about very specific information, e.g. flight
times or to perform a transaction, e.g. buy a ticket
• Also used by people with disabilities
– e.g. speech recognition word processors, page scanners, web readers,
home control systems
Lecture #6 IID_Interaction Design 124
Have speech interfaces come of age?
Lecture #6 IID_Interaction Design 125
Get me a human operator!
• Most popular use of speech interfaces currently is for call routing
• Caller-led speech where users state their needs in their own words
– e.g. “I’m having problems with my voice mail”
• Idea is they are automatically forwarded to the appropriate service
• What is your experience of speech systems?
Lecture #6 IID_Interaction Design 126
Format
• Directed dialogs are where the system is in control of the conversation
• Ask specific questions and require specific responses
• More flexible systems allow the user to take the initiative:
– e.g. “I’d like to go to Paris next Monday for two weeks.”
• More chance of error, since caller might assume that the system is like a human
• Guided prompts can help callers back on track
– e.g. “Sorry I did not get all that. Did you say you wanted to fly next Monday?”
Lecture #6 IID_Interaction Design 127
Research and design issues
• How to design systems that can keep conversation on track
– help people navigate efficiently through a menu system
– enable them to easily recover from errors
– guide those who are vague or ambiguous in their requests for information
or services
• Type of voice actor (e.g. male, female, neutral, or dialect)
– do people prefer to listen to and are more patient with a female or male
voice, a northern or southern accent?
Lecture #6 IID_Interaction Design 128
10. Pen
• Enable people to write, draw, select, and move objects at an interface
using lightpens or styluses
– capitalize on the well-honed drawing skills developed from childhood
• Digital pens, e.g. Anoto, use a combination of ordinary ink pen with
digital camera that digitally records everything written with the pen on
special paper
Lecture #6 IID_Interaction Design 129
Digital Ink
Lecture #6 IID_Interaction Design 130
Figure 6.22 Microsoft's digital ink in action showing how it can be used to annotate a scientific diagram Source: Reproduced by permission of Dennis Groth.
Pros and cons
• Allows users to quickly and easily annotate existing documents
• Can be difficult to see options on the screen because a user’s hand
can occlude part of it when writing
• Can have lag and feel clunky
Lecture #6 IID_Interaction Design 131
11. Touch
• Touch screens, such as walk-up kiosks, detect the presence and
location of a person’s touch on the display
• Multi-touch support a range of more dynamic finger tip actions, e.g.
swiping, flicking, pinching, pushing and tapping
• Now used for many kinds of displays, such as Smartphones, iPods,
tablets and tabletops
Lecture #6 IID_Interaction Design 132
Research and design issues
• More fluid and direct styles of interaction involving freehand and pen-based
gestures
• Core design concerns include whether size, orientation, and shape of touch
displays effect collaboration
• Much faster to scroll through wheels, carousels and bars of thumbnail images or
lists of options by finger flicking
• More cumbersome, error-prone and slower to type using a virtual keyboard on a
touch display than using a physical keyboard
Lecture #6 IID_Interaction Design 133
Research and design issues
• Will finger-flicking, swiping,
stroking and touching a screen
result in new ways of
consuming, reading, creating
and searching digital content?
Lecture #6 IID_Interaction Design 134
Figure 6.23 The Swype interface developed for mobile touch displays Source: Reproduced from http://www.geek.com/articles/mobile/nuances-t9-trace-virtual-keyboard-allows-you-to-swipe-rather-than-type-20100323/technology/.
12. Air-based gestures
• Uses camera recognition, sensor and computer vision techniques
– can recognize people’s body, arm and hand gestures in a room
– systems include Kinect
• Movements are mapped onto a variety of gaming motions, such as
swinging, bowling, hitting and punching
• Players represented on the screen as avatars doing same actions
Lecture #6 IID_Interaction Design 135
Xbox Kinect
Lecture #6 IID_Interaction Design 136
Figure 6.24 Microsoft's Xbox Kinect comprising an RGB camera for facial recognition plus video capturing, a depth sensor (an infrared projector paired with a monochrome camera) for movement tracking, and downward-facing mics for voice recognition Source: ©PA Images.
Home entertainment
• Universal appeal
• young children, grandparents,
professional gamers,
technophobes
Lecture #6 IID_Interaction Design 137
Figure 6.25 Touchless gesturing in the operating theater Source: Courtesy of Kenton O'Hara, Microsoft.
Gestures in the operating theatre
• A touchless system that
recognizes gestures
• surgeons can interact with and
manipulate MRI or CT images
• e.g. two-handed gestures for
zooming and panning
Lecture #6 IID_Interaction Design 138
Figure 6.26 The MusicJacket prototype with embedded actuators that nudge the player
Research and design issues
• How does computer recognize and delineate user’s gestures?
– Deictic and hand waving
• Does holding a control device feel more intuitive than controller free
gestures?
– For gaming, exercising, dancing
Lecture #6 IID_Interaction Design 139
13. Haptic
• Tactile feedback
– applying vibration and forces to a person’s body, using actuators that are
embedded in their clothing or a device they are carrying, such as a
smartphone
• Can enrich user experience or nudge them to correct error
• Can also be used to simulate the sense of touch between remote
people who want to communicate
Lecture #6 IID_Interaction Design 140
Realtime vibrotactile feedback
• Provides nudges when playing
incorrectly
• Uses motion capture
• Nudges are vibrations on arms
and hands
Lecture #6 IID_Interaction Design 141
Research and design issues
• Where best to place actuators on body
• Whether to use single or sequence of ‘touches’
• When to buzz and how intense
• How does the wearer feel it in different contexts?
• What kind of new smartphones/smart-watches apps can use
vibrotactile creatively?
– e.g. slow tapping to feel like water dropping that is meant to indicate it is
about to rain and heavy tapping to indicate a thunderstorm is looming
Lecture #6 IID_Interaction Design 142
14. Multi-modal
• Meant to provide enriched and complex user experiences
– multiplying how information is experienced and detected using different
modalities, i.e. touch, sight, sound, speech
– support more flexible, efficient, and expressive means of human–
computer interaction
– Most common is speech and vision
Lecture #6 IID_Interaction Design 143
Research and design issues
• Need to recognize and analyse speech, gesture, and eye gaze
• What is gained from combining different input and outputs
• Is talking and gesturing, as humans do with other humans, a natural
way of interacting with a computer?
Lecture #6 IID_Interaction Design 144
15. Shareable
• Shareable interfaces are designed for more than one person to use
– provide multiple inputs and sometimes allow simultaneous input by co-
located groups
– large wall displays where people use their own pens or gestures
– interactive tabletops where small groups interact with information using
their fingertips
– e.g. DiamondTouch, Smart Table and Surface
Lecture #6 IID_Interaction Design 145
A smartboard
Lecture #6 IID_Interaction Design 146
Figure 6.27 (a) A SmartBoard in use during a meeting and (b) Mitsubishi's interactive tabletop interface, where collocated users can interact simultaneously with digital content using their fingertips Source: (a) ©2006 SMART Technologies Inc. Used with permission. (b) Image courtesy of Mitsubishi Electric Research Labs.
(a)
DiamondTouch Tabletop
Lecture #6 IID_Interaction Design 147
Figure 6.27 (a) A SmartBoard in use during a meeting and (b) Mitsubishi's interactive tabletop interface, where collocated users can interact simultaneously with digital content using their fingertips Source: (a) ©2006 SMART Technologies Inc. Used with permission. (b) Image courtesy of Mitsubishi Electric Research Labs.
(b)
Roomware Furniture
Lecture #6 IID_Interaction Design 148
Figure 6.28 Roomware furniture Source: By permission of AMBIENTE.
Advantages
• Provide a large interactional space that can support flexible group
working
• Can be used by multiple users
– Can point to and touch information being displayed
– Simultaneously view the interactions and have same shared point of reference
as others
• Can support more equitable participation compared with groups using
single PC
Lecture #6 IID_Interaction Design 149
Research and design issues
• More fluid and direct styles of interaction involving freehand and pen-based
gestures
• Core design concerns include whether size, orientation, and shape of the display
have an effect on collaboration
• Horizontal surfaces compared with vertical ones support more turn-taking and
collaborative working in co-located groups
• Providing larger-sized tabletops does not improve group working but encourages
more division of labor
Lecture #6 IID_Interaction Design 150
16. Tangible
• Type of sensor-based interaction, where physical objects, e.g., bricks,
are coupled with digital representations
• When a person manipulates the physical object/s it causes a digital
effect to occur, e.g. an animation
• Digital effects can take place in a number of media and places or can
be embedded in the physical object
Lecture #6 IID_Interaction Design 151
Examples
• Chromarium cubes
– when turned over digital animations of color are mixed on an adjacent wall
– faciliates creativity and collaborative exploration
• Flow Blocks
– depict changing numbers and lights embedded in the blocks
– vary depending on how they are connected together
• Urp
– physical models of buildings moved around on tabletop
– used in combination with tokens for wind and shadows -> digital shadows surrounding
them to change over time
Lecture #6 IID_Interaction Design 152
Benefits
• Can be held in both hands and combined and manipulated in ways not
possible using other interfaces
– allows for more than one person to explore the interface together
– objects can be placed on top of each other, beside each other, and inside each
other
– encourages different ways of representing and exploring a problem space
• People are able to see and understand situations differently
– can lead to greater insight, learning, and problem-solving than with other kinds
of interfaces
– can facilitate creativity and reflection
Lecture #6 IID_Interaction Design 153
VoxBox
• A tangible system that gathers opinions at events through playful and
engaging interaction (Goldsteijn et al, 2015)
Lecture #6 IID_Interaction Design 154
Figure 6.29 VoxBox – Front and back of the tangible machine questionnaire Source: Golsteijn, C., Gallacher, S., Koeman, L., Wall, L., Andberg, S., Rogers, Y. and Capra, L. (2015) VoxBox: a Tangible Machine that Gathers Opinions from the Public at Events. In Proc. of TEI’ 2015. ACM.
Research and design issues
• Develop new conceptual frameworks that identify novel and specific features
• The kind of coupling to use between the physical action and digital effect
– If it is to support learning then an explicit mapping between action and effect is critical
– If it is for entertainment then can be better to design it to be more implicit and
unexpected
• What kind of physical artifact to use
– Bricks, cubes, and other component sets are most commonly used because of flexibility
and simplicity
– Stickies and cardboard tokens can also be used for placing material onto a surface
Lecture #6 IID_Interaction Design 155
17. Augmented and mixed reality
• Augmented reality - virtual representations are superimposed on
physical devices and objects
• Mixed reality - views of the real world are combined with views of a
virtual environment
• Many applications including medicine, games, flying, and everyday
exploring
Lecture #6 IID_Interaction Design 156
Examples
• In medicine
– virtual objects, e.g. X-rays and scans, are overlaid on part of a patient’s
body
– aid the physician’s understanding of what is being examined or operated
• In air traffic control
– dynamic information about aircraft overlaid on a video screen showing
the real planes, etc. landing, taking off, and taxiing
– Helps identify planes difficult to make out
Lecture #6 IID_Interaction Design 157
An augmented map
Lecture #6 IID_Interaction Design 158
Figure 6.30 An augmented map showing the flooded areas at high water level overlaid on the paper map. The handheld device is used to interact with entities referenced on the map Source: Reproduced with permission.
Top Gear James May in AR
• Appears as a 3D character to act as personal tour guide at Science
Museum
Lecture #6 IID_Interaction Design 159
Figure 6.31 James May appearing in 3D Augmented Reality Source: http://www.wired.com/2012/04/top-gear-host-narrates-museum-exhibits-as-augmented-reality-avatar/. Roberto Baldwin/Wired/©Conde Nast
James May Science Stories
Lecture #1 IID_Interaction Design 160
- Making of the Modern World at Science Museum 2013
https://vimeo.com/70035191
VR Experience at the Museum
Lecture #6 IID_Interaction Design 161
http://australianmuseum.net.au/landing/vr-experiences/
Research and design issues
• What kind of digital augmentation?
– When and where in physical environment?
– Needs to stand out but not distract from ongoing task
– Need to be able to align with real world objects
• What kind of device?
– Smartphone, head up display or other?
Lecture #6 IID_Interaction Design 162
18.Wearables
• First developments were head- and eyewear-mounted cameras that
enabled user to record what was seen and to access digital information
• Since, jewellery, head-mounted caps, smart fabrics, glasses, shoes, and
jackets have all been used
– provide the user with a means of interacting with digital information while on
the move
• Applications include automatic diaries, tour guides, cycle indicators and
fashion clothing
Lecture #6 IID_Interaction Design 163
Google Glass: short-lived
• What were the pros and cons?
Lecture #6 IID_Interaction Design 164
Figure 6.32 Google Glass Source: https://www.google.co.uk/intl/en/glass/start/. Google and the Google logo are registered trademarks of Google Inc., used with permission.
Talking Shoe
Lecture #6 IID_Interaction Design 165
https://www.thinkwithgoogle.com/campaigns/the-beatboxing-banner-and-the-talking-shoe.html
Research and design issues
• Comfort
– needs to be light, small, not get in the way, fashionable, and preferably hidden in the
clothing
• Hygiene
– is it possible to wash or clean the clothing once worn?
• Ease of wear
– how easy is it to remove the electronic gadgetry and replace it?
• Usability
– how does the user control the devices that are embedded in the clothing?
Lecture #6 IID_Interaction Design 166
19. Robots and drones
• Four types of robot
– remote robots used in hazardous settings
– domestic robots helping around the house
– pet robots as human companions
– sociable robots that work collaboratively with humans, and communicate
and socialize with them – as if they were our peers
Lecture #6 IID_Interaction Design 167
Advantages
• Pet robots are assumed to have therapeutic qualities, helping to
reduce stress and loneliness
• Remote robots can be controlled to investigate bombs and other
dangerous materials
Lecture #6 IID_Interaction Design 168
Robots
Lecture #6 IID_Interaction Design 169
Figure 6.33 Left: Mel, the penguin robot, designed to host activities; right: Japan's Paro, an interactive seal, designed as a companion, primarily for the elderly and sick children Source: (left) Image courtesy of Mitsubishi Electric Research Labs. (right) Courtesy of Parorobots.com.
Drones
• Unmanned aircraft that are controlled remotely and used in a number of contexts
– e.g. entertainment, such as carrying drinks and food to people at festivals and parties;
– agricultural applications, such as flying them over vineyards and fields to collect data
that is useful to farmers
– helping to track poachers in wildlife parks in Africa
• Can fly low and and stream photos to a ground station, where images can be
stitched together into maps
• Can be used to determine the health of a crop or when it is the best time to harvest
the crop
Lecture #6 IID_Interaction Design 170
OppiKoppi Beer Drone Tech
Lecture #6 IID_Interaction Design 171
http://www.oppikoppi.co.za/
Drone in vineyard
Lecture #6 IID_Interaction Design 172
Figure 6.34 A drone being used to survey the state of a vineyard Source: Courtesy of Discover Sonoma County Wine http://www.latimes.com/business/la-fi-drones-agriculture-20140913-story.html#page=1
Research and design issues
• How do humans react to physical robots designed to exhibit behaviors (e.g. making
facial expressions) compared with virtual ones?
• Should robots be designed to be human-like or look like and behave like robots that
serve a clearly defined purpose?
• Should the interaction be designed to enable people to interact with the robot as if it
was another human being or more human-computer-like (e.g. pressing buttons to issue
commands)?
• Is it acceptable to use unmanned drones to take a series of images or videos of fields,
towns, and private property without permission or people knowing what is happening?
Lecture #6 IID_Interaction Design 173
20. Brain-computer interfaces
• Brain–computer interfaces (BCI) provide a communication pathway
between a person’s brain waves and an external device, such as a
cursor on a screen
• Person is trained to concentrate on the task, e.g. moving the cursor
• BCIs work through detecting changes in the neural functioning in the
brain
• BCIs apps:
– Games
– enable people who are paralysed to control robots
Lecture #6 IID_Interaction Design 174
Brainball game
Lecture #6 IID_Interaction Design 175
Figure 6.35 The Brainball game using a brain–computer interface Source: “Brainball” from The Interactive Institute. Reproduced with permission.
Lecture #6 IID_Interaction Design 176
Paralysed woman moves robot with her mind
Lecture #6 IID_Interaction Design 177
Figure 2 Participant S3 drinking from a bottle using the DLR robotic arm. http://www.nature.com/nature/journal/v485/n7398/full/nature11076.html
Which interface?
• Is multimedia better than tangible interfaces for learning?
• Is speech as effective as a command-based interface?
• Is a multimodal interface more effective than a monomodal interface?
• Will wearable interfaces be better than mobile interfaces for helping
people find information in foreign cities?
• Are virtual environments the ultimate interface for playing games?
• Will shareable interfaces be better at supporting communication and
collaboration compared with using networked desktop PCs?
Lecture #6 IID_Interaction Design 178
Which interface?
• Will depend on task, users, context, cost, robustness, etc.
• Mobile platforms taking over from PCs
• Speech interfaces also being used much more for a variety of commercial services
• Appliance and vehicle interfaces becoming more important
• Shareable and tangible interfaces entering our homes, schools, public places, and
workplaces
Lecture #6 IID_Interaction Design 179
Summary
• Many innovative interfaces have emerged post the WIMP/GUI era,
including speech, wearable, mobile, brain and tangible
• Raises many design and research questions to decide which to use
– e.g. how best to represent information to the user so they can carry out
ongoing activity or task
• New interfaces that are context-aware or monitor raise ethical issues
concerned with what data is being collected and what it is used for
Lecture #6 IID_Interaction Design 180
Mid Term Examination
• Time & Date
– 00:00 am – 11:59 pm on 19th April
• Take Home & Open Book Exam
– Includes textbook chapter 1 – 6 that we have read during the half of
semester.
– The test questions will be posted on the class blog, and facebook page
until 11:00 pm on 18th April.
– Please submit the answers until 11:59 pm, Wendesday,19th April.
– Late submission will reduce your mark.
Lecture #6 IID_Interaction Design 181