hci-5 evaluation techniques

8/4/2019 HCI-5 Evaluation Techniques

1/40

IAS2223

HUMAN-COMPUTER INTERACTION

Topic 5

Evaluation Techniques


2/40

EVALUATION TECHNIQUES

Evaluation

y tests usability, functionality and acceptability of an

interactive system

y occurs in laboratory, field and/or in collaboration

with users

y evaluates both design and implementation

y should be considered at all stages in the design life

cycle


3/40

GOALS OF EVALUATION

assess extent of system functionality

assess effect of interface on user

identify specific problems


4/40

EVALUATION THROUGH

EXPERT ANALYSIS

Cognitive Walkthrough

Heuristic Evaluation

Model-based evaluation


5/40

COGNITIVE WALKTHROUGH

Proposed by Polson et al.

y Require a details review of a sequence of actions

y evaluates design on how well it supports user in

learning tasky usually performed by expert in cognitive

psychology

y expert walks though design to identify

potential problems using psychologicalprinciples

y Main focus: learning through exploration

y forms used to guide analysis


6/40

COGNITIVE WALKTHROUGH

Requirements:

i. A specification / prototype of the system

ii. A description of task user is to perform

on the system

iii. A written list of actions needed to

complete the task

iv. An indication about users experience& knowledge


7/40

COGNITIVE WALKTHROUGH (CTD)

For each task walkthrough considers

y what impact will interaction have

on user?

y what cognitive processes are

required?

y what learning problems may

occur?


8/40

IN EACH ACTION PERFORM:

Analysis focuses on goals and knowledge: does

the design lead the user to generate the correct

goals?

Will users see that the action is available? Does user know that the action is the one that

they needed?

After action is taken, will user understand the

feedback they get?


9/40

HEURISTIC EVALUATION

Heuristics - guideline / general principlesthat can guide a decision that has alreadybeen made.

usability criteria (heuristics) are identified

design examined by experts to see if these areviolated

Example heuristicsy system behaviour is predictable

y system behaviour is consistent

y feedback is provided

Heuristic evaluation `debugs' design.


10/40

NIELSENS 10 HEURISTICS:

Visibility of system status

Match between system & real world

User control & freedom

Consistency & standardsError prevention

Recognition rather than recall

Flexibility & efficiency of use

Aesthetic & minimal design

Help users recognize, diagnose & recoverfrom errors

Help & documentation


11/40

MODEL-BASED EVALUATION

Results from the literature used to support orrefute parts of design.

Care needed to ensure results aretransferable to new design.

Cognitive models used to filter design optionse.g. GOMS (goals, operators, methods, selections)

prediction of user performance.

Design rationale can also provide usefulevaluation information


12/40

EVALUATING THROUGH USER

PARTICIPATION


13/40

LABORATORY STUDIES

Advantages:y specialist equipment available

y uninterrupted environment

Disadvantages:y lack of context

y difficult to observe several users cooperating

Appropriatey if system location is dangerous or impractical for

constrained single user systems to allow controlledmanipulation of use


14/40

FIELD STUDIES

Advantages:y natural environment

y context retained

y Interaction occur in actual use

Disadvantages:y distractions

y noise

Appropriatey where context is crucial for longitudinal studies


15/40

Evaluating Implementations

Requires an artefact:

simulation, prototype, full implementation


16/40

EMPIRICAL METHODS:

EXPERIMENTAL EVALUATION

controlled evaluation of specific aspects of

interactive behaviour

evaluator chooses hypothesis to be tested

a number of experimental conditions are

considered which differ only in the value of

some controlled variable.

changes in behavioural measure are

attributed to different conditions


17/40

EXPERIMENTAL FACTORS

Participants

y who representative, sample size

Variables

y things to modify and measure Hypothesis

y what youd like to show

Experimental design

y

how you are going to do it


18/40

VARIABLES

independent variable (IV)

characteristic changed to produce different

conditions

e.g. interface style, number of menu items

dependent variable (DV)

characteristics measured in the experiment

e.g. time taken, number of errors.


19/40

HYPOTHESIS

prediction of outcomey framed in terms of IV and DV

e.g. error rate will increase as font size decreases

null hypothesis:y states no difference between conditions

y aim is to disprove this

e.g. null hyp. = no change with font size


20/40

EXPERIMENTAL DESIGN

within subjectsy each subject performs experiment under each

condition.

between subjectsy each subject performs under only one condition


21/40

ANALYSIS OF DATA

Before you start to do any statistics:y look at data

y save original data

Choice of statistical technique depends ony type of data

y information required

Type of datay discrete - finite number of values

y continuous - any value


22/40

ANALYSISSTATISTICAL TECHNIQUES

parametricy ANOVA (analysis of variance)

y robust

y powerful

non-parametricy do not assume normal distribution

y less powerful

y

more reliable

contingency tabley classify data by discrete attributes

y count number of data items in each group


23/40

ANALYSIS OF DATA(CONT.)

What information is required?

y is there a difference?

y how big is the difference?

y

how accurate is the estimate?

Parametric and non-parametric tests mainly

address first of these


24/40


25/40

SUBJECT GROUPS

larger number of subjects

more expensive

longer time to `settle down even more variation!

difficult to timetable

so often only three or four groups


26/40

DATAGATHERING

several video cameras

+ direct logging of application

problems:y synchronisation

y sheer volume!

one solution:

y record from each perspective


27/40


28/40

THINKALOUD

user observed performing task

user asked to describe what he is doing andwhy, what he thinks is happening etc.

Advantagesy simplicity - requires little expertise

y can provide useful insight

y can show how system is actually use

Disadvantagesy subjective

y selective

y act of describing may alter task performance


29/40

COOPERATIVE EVALUATION

variation on think aloud

user collaborates in evaluation

both user and evaluator can ask each other

questions throughout

Additional advantages

y less constrained and easier to use

y user is encouraged to criticize systemy clarification possible


30/40

PROTOCOL ANALYSIS

Methods for recording user actions:

paper and pencil cheap, limited to writing speed

audio good for think aloud, difficult to match with otherprotocols

video accurate and realistic, needs special equipment,obtrusive

computer logging automatic and unobtrusive, largeamounts of data difficult to analyze

user notebooks coarse and subjective, useful insights, goodfor longitudinal studies

Mixed use in practice.

audio/video transcription difficult and requires skill.

Some automatic support tools available


31/40

AUTOMATED ANALYSIS EVA(EXPERIMENTALVIDEOANNOTATOR)

Workplace project

Post task walkthrough

y user reacts on action after the event

y used to fill in intention

Advantages

y analyst has time to focus on relevant incidents

y avoid excessive interruption of task

Disadvantagesy lack of freshness

y may be post-hoc interpretation of events


32/40

POST-TASK WALKTHROUGHS

transcript played back to participant for

comment

y immediatelyp fresh in mind

y delayedp evaluator has time to identify questions

useful to identify reasons for actions and

alternatives considered

necessary in cases where think aloud is not

possible


33/40

QUERY TECHNIQUES

Interviews

Questionnaires


34/40

INTERVIEWS

analyst questions user on one-to -one basisusually based on prepared questions

informal, subjective and relatively cheap

Advantagesy can be varied to suit context

y issues can be explored more fully

y can elicit user views and identify unanticipated

problemsDisadvantages

y very subjective

y time consuming


35/40

QUESTIONNAIRES

Set of fixed questions given to users

Advantages

y quick and reaches large user group

y can be analyzed more rigorously

Disadvantages

y less flexible

y less probing


36/40

QUESTIONNAIRES (CTD)

Need careful design

y what information is required?

y how are answers to be analyzed?

Styles of question

y general

y open-ended

y scalary multi-choice

y ranked


37/40

PHYSIOLOGICAL METHODS

Eye tracking

Physiological measurement


38/40

EYE TRACKING

head or desk mounted equipment tracks the

position of the eye

eye movement reflects the amount of

cognitive processing a display requiresmeasurements include

y fixations: eye maintains stable position. Number

and duration indicate level of difficulty with

display

y saccades: rapid eye movement from one point of

interest to another

y scan paths: moving straight to a target with a

short fixation at the target is optimal


39/40


40/40

CHOOSINGAN EVALUATION METHOD

Factors distinguish evaluation techniques:

when in process: design vs. implementation

style of evaluation: laboratory vs. field

how objective: subjective vs. objective

type of measures: qualitative vs. quantitative

level of information: high level vs. low level

level of interference: obtrusive vs. unobtrusiveresources available: time, subjects,

equipment, expertise

hci-5 evaluation techniques

Documents