Paper Prototype Evaluation

Evaluation

• Well, before addressing the evaluation of paper prototypes, perhaps we should consider reasoning upon what we should evaluate, right?– Let’s then take a look into concepts such as

usefulness, usability and user experience

Usefulness

• Usefulness is a crucial quality of any product or service– Concerns the degree to which a product enables a

user to achieve his or her goals, and is an assessment of the user’s willingness to use the product or service at all• Without usefulness, other measures make little or no

sense, because it will just sit on the shelf

• We name this the fundamental quality of a product or service

Usability

• According to the ISO standard– Usability is the extent to which a product or

service can be used by specific users to achieve predefined goals in a specified context of use

– The standard defines three main usability dimensions:• Effectiveness,• Efficiency, and• Satisfaction

Usability

• Anyway, when looking into usability, we normally account for:– Effectiveness– Efficiency– Satisfaction – Learnability – Memorability

• We name these the pragmatic qualities of a product or service

Usability

• Common reason for the delivery of less usable products are– System focused process– Poorly integrated teams– Design and development mismatches– And of course, designing usable products is

difficult• That’s what we are pushing paper prototyping in this

moment of the process

User Experience

• According to its ISO standard, user experience is a person’s perception and responses that result from the use or anticipated use of a product or service

User Experience

• User experience subsumes usability and includes includes– emotions, beliefs, preferences, perceptions,

physical and psychological responses, behaviors and accomplishments that occur before, during and after use

• We name these the effected of the hedonic, or pleasure related, qualities of a product or service

Evaluation

• As stating in the beginning, before addressing the evaluation of paper prototypes, we should consider reasoning upon what we should evaluate– And now that we briefly cover usefulness, usability

and user experience, I believe it’s fair to say that we should target going as far as usability

Evaluation

• In general, product and service evaluation can be formative or summative

• Paper prototype evaluation is mostly formative

Evaluation

• Paper prototype evaluation focus on…– The most significant issues preventing users from

accomplishing their goals– What works and what do users find frustrating– What are the most common errors or mistakes

users are making– Assessing the improvements being made from one

design interaction to the next– Identifying issues that are expected to remain even

when the product is launched

Evaluation

• Today we look into– Inspection methods• These are methods where an expert evaluator inspects

a product or service

– Testing methods• These are methods where products and services are

evaluated by testing them on real users

Inspection methods

• There are also several but we will address:– Cognitive walk-through– Heuristic evaluation

Cognitive walk-through

• The purpose is to verify if the paper prototype actually allows the fulfillment of the selected user stories– This is a within team activity that ensures that

your prototype complies with the identified user stories

Cognitive walk-through

• Designers and developers of the product or service then walk through the steps as a group, asking themselves a set of questions at each step– Data is gathered during the walk-through, and

afterwards a report of potential issues is compiled– Finally the evaluated proposition is redesigned to

address the issues identified

Heuristic evaluation

• This is a type of evaluation ideally carried out by an expert.– It specifically involves evaluators examining the

design and judging its compliance with recognized principles, such as Jakob Nielsen’s 10 usability heuristics• These evaluation methods are now widely taught and

practiced in the new media sector, where products and services are often designed in a short space of time on a budget that may restrict the amount of money available to provide for other types of interface testing

Testing methods

• There are several but we will address:– Co-discovery– Wizard of Oz

Co-discovery

• Two users attempt to perform tasks together while being observed– They are to help each other in the same manner

as they would if they were working together to accomplish a common goal using the product

– They are encouraged to explain what they are thinking about while working on the tasks

Co-discovery

• The designers and developers should refrain from explaining the design decisions and rather focus on getting the most of the pair of users tacking with your prototype– Note taking is fundamental and you should run

this kind of test until no significant additional information is feed back into the design process

Wizard of OZ

• This is a testing approach built upon a paper device instead of using a working technological artifact– This king of testing involves systematic observation

under controlled conditions to determine how well people can use a product or service

– Rather than showing users a rough draft and asking, Do you understand this?, this kind of testing involves watching people trying to use something for its intended purpose

Wizard of OZ

• Setting up such a test involves asking the test subjects to recreate a set of user stories after being introduced to the underlying scenario– For example, to test the attachment function of an

e-mail program, a scenario would describe a situation where a person needs to send an e-mail attachment, and ask him or her to undertake this task

– The aim is to observe how people function in a realistic manner, so that developers can see problem areas, and what people like.

Testing methods

• This kind of evaluation should be repeated until no significant added value comes from bring in an additional subject– The usual number is 5, but other claim otherwise

Pre and post-tests

• When applying testing methods, pre-test and post-test questionnaires are also used to gather feedback on the product being tested– A common questionnaire is the 25 years old

System Usability Scale

System Usability Scale

Evaluation protocol

• But these methods are useless without an adequate evaluation protocol

• While designing your evaluation protocol, you should take into account…– A mixed selection of inspection and testing

methods;– The user stories supported by your prototype; and– The affordances of your paper prototype.

Evaluation protocol

• Examples are provided in the companion blog but please note that…– In all cases, evaluation protocols should be piloted

to ensure that once they are being applied, you are actually focusing in assessing the prototype and not on solving evaluation protocol issues

Now it’s up to you to make it happen

paperprototypingateia.wordpress.com