seng 310 l13 evaluation with users - electrical aalbu/seng310_2010/seng 310 l13 evaluation with...

Download SENG 310 L13 Evaluation with users - Electrical aalbu/seng310_2010/SENG 310 L13 Evaluation with users.pdfآ 

Post on 17-Oct-2019




0 download

Embed Size (px)


  • 2

  • 3

  • 4

    I would like you to focus on the role users play in software redesign; what type of evaluation techniques are used

    From 1:16 til 4:58 What types of usability studies does she mention:

    - usability; longitudinal; surveys; service quality monitor data; watch sequences of commands that the users use.

    - “user is boss” – about product mastering: features that are not used.

    - Consistency and standards: moving away from menus and toolbars: information organization must be different when dealing with 1500 commands

  • 5

    The Observer side is where the Usability Engineer and other observers may view the study. The one-way mirror allows observers to clearly see the participant side during the study, while minimizing any distractions for the participant.

    The wall and the one-way mirror are sound proof so observers can discuss design ideas in a normal tone without disturbing the participant.

    The engineer can communicate with the participant via the microphone on the desk, which is activated by a button on the microphone's base. The participant can only hear what is said on the observer side when the engineer activates the microphone. This reduces the likelihood that participants will be distracted from discussions by the engineers during the test.

    This type of set-up is useful for testing advanced prototypes.

  • 6

    The formative evaluation is finding usability problems in order to fix them in the next design iteration.

    - doesn’t need a full working implementation, and it can be done on a variety of prototypes

  • 7

    A key problem with formative evaluation is that you have to control too much. Running a test in a lab environment on tasks of your invention may not tell you enough about how well your interface will work in a real context on reals tasks. A field study can answer these questions, by actually deploying a working implementation to real users and then going out to the user’s real environment and observing how they use it.

    A field study is mostly about watching the user:

  • 8

    A third kind of user test is a controlled experiment, whose goal is to test a quantifiable hypothesis about one or more interfaces (it is not necessarily a comparison test)

    Controlled experiments happen under carefully controlled conditions about carefully designed tasks – often more carefully chosen than formative evaluation tasks.

    Hypotheses can only be tested by quantitative measurements of usability (such as time elapsed, number of errors, or subjective satisfaction). We’ll talk about how to design controlled experiments in the next lecture.

  • 9

  • 10

    Undue Persuasion

    The design of persuasive technologies presents some

    ethical issues for HCI designers. [3] highlights ethical

    principles for the design of persuasive technologies. Here

    are two examples from their list: · Rule VII: “Persuasive technologies must not misinform

    in order to achieve their persuasive end.”

    · Rule VIII: “The creators of a persuasive technology should never seek to persuade a person or persons of

    something they themselves would not consent to be

    persuaded to do.”

    Persuasive technology is broadly defined atechnology that is designed to change attitudes or behaviors of the users through persuasion and social influence, but not through coercion (Fogg 2002).

    Example: digital health coaching

  • 11

    HCI professionals face ethical issues every day. Here are some examples that could form the basis for a set of case


    Answer: You can’t give any answer that reveals or even narrows down the identity of an interviewee, no matter

    who is asking - a promise is a promise.

  • 12

  • 13

  • 14

    Most user testing in HCI has minimal physical or psychological risk to the user.

    But user testing does put psychological pressure on the user. The user sits in the spotlight, asked to perform unfamiliar tasks on an unfamiliar (and possibly bad) interface, in front of an audience of strangers (at least one experimenter, several observers, and a video camera).

    It is natural to feel some performance anxiety, or stage fright. - Regard the test as a psychology test, or worse, as an IQ test. - Worried about getting a bad score - They self-esteem may suffer, especially if they blame problems they have on themselves instead than on the user interface.

    Accumulation of mistakes during a user testing session:

    - The originally scheduled user didn’t show up, so they just pulled an employee out of the hallway to do the test

    - This employee happened to be on her first day at the job - They did not tell her what the session was about - She knew nothing about the UI and about the domain (she wasn’t in the target population)

    - The observers in the room hadn’t been told how to behave (ie shut up and watch their body language)

    - One of the observers was its boss - The tasks hadn”t been pilot tested, and the first one was impossible - When she started struggling with the first task, the designers realized that this task was impossible and burst into laughter at their own stupidity, not her’s.

  • 15

  • 16

    Participants shouldn't feel like they're locked into completing tasks

  • 17

  • 18

  • 19

  • 20

    Excellent at identifying gross design/interface problems

  • 21

    Thinking aloud=verbalizing their interactions with the system. Thinking aloud gives you (the observer) a window into their thought processes, so that you can understand what they are trying to do and what they expect.

    - You can also estimate the gulf of execution and the gulf of evaluation

    Problems: - Feels weird for most people. - It can alter the user’s behaviour, making the user more deliberate and careful, and sometimes disrupting concentration.

    - Conversely, when a task gets harder and the user gets absorbed in it, they may go mute, forgetting to think aloud. One of the facilitator’s roles is to encourage the user to think aloud.

  • 22

    • Variant: Co-discovery learning

    - use semi-knowledgeable “coach” and naive subject together

    - make naive subject use the interface

    • results in

    - naive subject asking questions - semi-knowledgeable coach responding

    - provides insights into thinking process of both

    beginner and intermediate users

    Problems: constructive interaction requires twice as many users, and can be adversely affected by social dynamics (if the two users don’t go along very well).

  • 23

    The facilitator = leader of the user test

    - Does the briefing - Gives tasks to the user - Generally serves as the voice of the development team throughout the test

    - One of the key job: make the user think aloud, usually by asking general questions

    - He also moves the session along. If the user is totally stuck on a task, the facilitator may progressively provide more help.

    - Only do this if you’ve already recorded the usability problem, and it seems unlikely that the users will get out of the tar pit themselves. So they need to get unstuck in order to move on to another part of the task that you want to test.

    - Keep in mind that once you explain something, you loose the chance to find out what the user would have done by themselves.

  • 24

    On the blackboard

    Critical incidents=events that strongly affect task performance or user satisfaction

    When the user is thinking aloud, and the facilitator is coaching the thinking aloud, any observers in the room should do the opposite: keeping quiet. Don’t offer any help, don’t attempt to explain the interface. Just watch: you’re trying to get a glimpse of how a typical user interacts with your interface. Since a typical user won’t have the system designer sitting next to them, you have to minimize your effect on the situation. It may be very hard for you to sit and watch someone struggle with a task, when the solution seems so obvious to you, but that’s how you learn the usability problems in your interface.

    Keep yourself busy by taking a lot of notes. What should you take notes about? As much as you can, but focus on critical incidents:

    Moments that strongly affect usability ( either in task performance, efficiency and error rate, or in user’s satisfaction). Most critical incidents are negative:

    - Pressing the wrong button is a critical incident - So is repeatedly trying the same feature to accomplish a task. - Positive: pleasant surprises should be also noted down. - Critical incidents give you a list of usability problems that you should focus on the next round of iterative design.

  • 25

    Paper notes are usually the best, although it may be hard to keep up; useful to have many observers taking notes Audio recording is obviously useful and necessary for capturing the thinking aloud. Audio/Video recording is good because it captures other details than thinkin