Midterm Stats

• Min: 16/38 (42%)

• Max: 36.5/38 (96%)

• Average: 29.5/36 (78%)

Evaluation 2: Choosing a Strategy

Objectives

For evaluation of a given application, you will be able to…

• Prioritize goals and specify usability requirements

• Decide what type of data you need to collect• Decide who should take part• Develop good tasks• Decide where and for how long• Justify your decisions for all of the above

Why bother?

• Tied to the usability engineering lifecycle

• Pre-design– investing in new expensive system requires proof of

viability

• Initial design stages– develop and evaluate initial

design ideas with the user

design

implementationevaluation

Why bother?• Iterative design

– does system behavior match the user’s task requirements?

– are there specific problems with the design?– what solutions work?

• Acceptance testing– verify that system meets

expected user performance criteria• 80% of 1st time customers will take 1-3 minutes to

withdraw $50 from the automatic teller

design

implementationevaluation

Experimental approachExperimenter controls all environmental factors

– study relations by manipulating independent variables– observe effect on one or more dependent variables

– Nothing else changes

File Edit View Insert

New

Open

Close

Save

File

Edit

View

Insert

New

Open

Close

Save

There is no difference in user performance (time and error rate) when selecting an item from a pull down or a pop-up menu of 4 items

Usability engineering approach

• Observe people using systems in simulated settings– people brought in to artificial setting that simulates

aspects of real world setting

– people given specific tasks to do

– observations / measures made as people do their tasks

– look for problem areas / successes

– good for uncovering ‘big effects’

Usability engineering approach

• Is the test result relevant to the usability of real products in real use outside of lab?

• Problems– non-typical users tested– non-typical tasks– different physical environment– different social context

• motivation towards experimenter vs motivation towards boss

• Partial Solution– use real users– task-centered system design tasks– environment similar to real situation

Conceptual model extraction• How?

– show the user static images of• the prototype or screens during use

– ask the user explain • the function of each screen element• how they would perform a particular task

• What?– Initial conceptual model

• how person perceives a screen the very first time it is viewed

– Formative conceptual model • How person perceives a screen after its been used for a while

• Value?– good for eliciting people’s understanding before & after use– poor for examining system exploration and learning

Why are you evaluating?: 5 E’s of Usability

• Effective• Efficient• Engaging• Error Tolerant• Easy To Learn

• But, you don’t always want to test all of these!

Why are you evaluating?: Identify your goals

• Identify your usability goals for the interface

• Then prioritize them to determine which you need to test

Who?

• Actual end users• Varied types of users

– Different user groups– Different ages– Different experience– Etc…

How Many Users?

• Depends on variability of users and how stringently you want to test.

• observing many users is expensive• but individual differences matter

– best user 10x faster than slowest

– best 25% of users ~2x faster than slowest 25%

Solution?• Textbook recommends starting with 5 from each user

group, adding more as needed.• Big problems identified with a few users, smaller

problems may need many.

Where?: Field vs. Lab

• Field studies – realistic setting– True to real life– Hard to arrange and do– Less control of variables

• Controlled studies - controlled environment– More control but less realistic– Often easier to set up, more

efficient

Validity• External validity

– confidence that results applies to real situations– usually good in natural settings

• Internal validity– confidence in our explanation of experimental results– usually good in experimental settings

• Trade-off: Natural vs Experimental– precision and direct control over experimental design

versus– desire for maximum generalizability in real life

situations

How long should a session last?

• Usually 30-90 minutes / session

• May be longer, but beware of fatigue and learning effects

• Sometimes need several sessions (e.g. for tasks that require substantial training)

What should the participants do?

• Think carefully about the tasks you ask people to do.

• Task should model real-world use.

• May also want to test rare / unusual situations

Key Points

• Plan - to ensure you are testing the right things with the right people.

• Consider the goals & usability requirements, data to collect, users, tasks, location, and length of the evaluation.