1ISE 412 - 2

User-centered design

Theme - “know thy user,” “honor thy user” Involves four major components:

Early focus on user and task Empirical measurement Iterative design Participatory design

2ISE 412 - 2

Norman’s Theory of Action

The World

Goals

Intention to act

Sequence of actions

Execution of actions

Interpreting the perceptions

Perceiving the state of the world

Evaluation of interpretations

3ISE 412 - 2

“Gulfs” of Evaluation and Execution

The World

Goals

Complexity, inappropriate model

Complexity, unknown alternatives

Complexity, poor design

Inappropriate feedback, mental models

Invisible states, time lags

Inappropriate mental models

Gulf of Execution Gulf of Evaluation

4ISE 412 - 2

Design to “bridge the gulfs”

The World

Goals

Evaluate the need/ability to affect the state

Understand what should be done

Perform the desired actions

Interpret data to understand what's happening

Get information about what's happening

Understand what the system state is/should be

How easy is it to ...? How easy is it to ...?

5ISE 412 - 2

The key

Understanding human behavior, cognitive processes, capabilities, and limitations

Understanding what people are doing, why, where, and under what conditions To explore further …

Norman, D.A. (2005) Human-Centered Design Considered Harmful, http://www.jnd.org/dn.mss/ human-centered_desig.html

6ISE 412 - 2

The Information Processing Model

from: Wickens, Lee, Liu, & Becker (2004) An Introduction to Human Factors Engineering (2nd ed.). New Jersey: Pearson Prentice Hall. pg.122

7ISE 412 - 2

HUMAN FACTORS RESEARCH What can we measure?

Attitude and motivation Preferences Knowledge Skills and abilities (physical and mental) Aptitude and potential to perform/succeed Emotion (happy, sad, mad, excited, tense) Physiological states Behavior Performance

8ISE 412 - 2

RESEARCH METHODS IN HUMAN FACTORS ENGINEERING

Basic vs Applied Definitions:

Basic Applied

9ISE 412 - 2

Dimensions of Research Methodologies:

Tradeoff: Experimental Control vs Real-World Relevance

Degree of Control: over conditions of observation over "treatment" (independent) variables

Degree to which behavioral (dependent) variables are representative of some larger population

Will the results "scale up" to complex real-life situations?

10ISE 412 - 2

Research Methodology “In a Nutshell”

Define problem and hypotheses What questions do you want to answer? What cause/effect relationships do you expect to see?

Specify the experimental plan Independent variables (Treatment):

Dependent variables (Effect):

Experimental design:

Specifics:

11ISE 412 - 2

Research Methodology “In a Nutshell” (cont.) Conduct the study

follow the plan you defined use a “pilot study” if you are unsure of your plan

Analyze the data Quantitative/objective data

Subjective data

Protocol analyses

12ISE 412 - 2

Research Methodology “In a Nutshell” (cont.) Draw conclusions

Were the hypotheses supported by the results of the experiment?

Statistically significant? (Usually use p = 0.05 or 0.01, but these are not carved in stone!)

Recall Type I and Type II errors … If not significant, are there interesting trends in the data? Interaction effects? (e.g., mouse vs keyboard confounded

with type of input required when filling out online forms) Can you say why?

Will effect the degree to which you can generalize your results.

Can you explain interaction effects, trends that are not statistically significant?

13ISE 412 - 2

An overview of experimental types:Laboratory Experiment Example: Carter's 1979 study of the effects of various display characteristics

on search time using college students in a dark quiet room.

Experimental Control Relevance

High

Low

14ISE 412 - 2

An overview of experimental types:Field Experiment / Quasi-Experiment Examples: (1) Comparing new ATC interface with existing using real air traffic

controllers in real-life settings (field experiment). (2)Comparing "New Math" to conventional instruction by assigning programs to different classrooms (quasi-experiment).

Experimental Control Relevance

Moderate

15ISE 412 - 2

An overview of experimental types:

Simulation Study Examples: Flight simulators, power plant control simulators.

Experimental Control Relevance

High

Low

16ISE 412 - 2

An overview of experimental types:Field Study (Case study) Examples: (1) flight crew task analyses conducted in-flight to

determine task allocation, workload, etc. prior to specification and design of automated systems; (2) investigators observation and modeling of satellite ground operations

Investigator observes and records real-life activities without any experimental manipulations. Typically, such a study seeks to articulate how and why some

behavioral phenomenon occurs.

Generally, should be the first step in investigating human interaction with complex systems.

17ISE 412 - 2

Descriptive Methods

Survey/Questionnaire Used to gauge attitudes, preferences, etc., often on a

numerical scale (e.g., 1=strongly disagree, ..., 5=strongly agree). Amenable to statistical analyses.

Problems: how to avoid biased questions

Respondents may not be totally truthful

18ISE 412 - 2

Descriptive Methods (cont.)

Interview Also used to gauge attitudes, preferences, etc.

Can delve into the reasons behind those attitudes.

Can be used with a questionnaire to clarify or expand on responses.

Problems: Same as with survey/questionnaire.

Tricky to analyze.

19ISE 412 - 2

Descriptive Methods

Concurrent Verbal Protocols While the subject performs a task (in a laboratory, field, or

simulation setting), he/she verbalizes- thoughts, reasons for actions, etc. This is often very illuminating in terms of identifying subject strategies, difficulties, etc. But difficult to analyze and may affect task performance itself.

Retrospective Verbal Protocols After the subject has performed a task (in a laboratory, field, or

simulation setting), he/she verbalizes thoughts, reasons for actions, etc. This may also be quite interesting, but again is difficult to analyze. Also this relies on the subject's memory, which is prone to error.

Also: critical incident technique, accident analysis.

20ISE 412 - 2

General Remarks: Before you do any experiment, you should have one or

more hypotheses in mind that you want to test or explore.

If possible, use multiple methods (e.g., field study and interviews and questionnaires). If you obtain converging evidence from a number of separate

sources, you can be more confident that the hypothesis is correct.

Consider reliability and validity issues: Reliability:

Construct Validity:

21ISE 412 - 2

General Remarks (cont.):

Reliability and validity issues (concl.): Internal Validity (for causal or explanatory studies):

External Validity:

Ethical Issues: Ethical treatment of human subjects

protection from mental and physical harm privacy informed consent

Code of Engineering Ethics

22ISE 412 - 2

Contextual Inquiry (Field Study +) A fact-based approach to

understanding the reality of users’ goals, processes, and tasks.

Puts systems designers directly in the “world” of the user to see, hear, feel, taste, smell, and better understand: the goals of the user the activities and specific behaviors

users engage in to achieve those goals the places in which they occur the tools, supplies, etc. that are used the artifacts in the environment and

their meanings etc.

Provides rich, context-specific data from which to develop a shared understanding of users that will guide design

23ISE 412 - 2

C.I. vs Field StudyField Study Contextual Inquiry

Typical Focus

(Traditionally) operators, their activities, tools, information requirements, decision making, work environment, etc., with emphasis on specific tasks or goals.

Interactions of people, environments, tools, and artifacts in specific workplaces or doing specific types of work, with emphasis on communication & cultural aspects.

Typical Approach

Researcher is an observer who tries not to interfere with task performance. Follow-up interviews are used to gather more info.

Researcher is a participant and the operator is considered a partner in the process. Questions may be asked during observations, with follow-up interviews as well.

Typical Product

Task analyses, physical models, information flow models.

Flow models, cultural models, physical models, sequence models, and artifact models.

24ISE 412 - 2

The C. I. Process(Source: Beyer, H. & Holtzblatt, K. (1998) Contextual Design: Defining Customer-Centered

Systems. San Diego: Academic Press)

Design the research Recruit participants Collect data

interview (video taping optional) introduction initial interview contextual interview wrap-up

research notes artifacts “homework”, prework, etc. if applicable

Analyze video, notes, visuals, artifacts, etc. Synthesize results

25ISE 412 - 2

Designing Contextual Studies: General Approach What will you be

investigating? specific activities, problems,

etc. environments target population

Script what questions will you ask in

the interview(s)? are there specific tasks or

problems you want to be sure to observe?

what follow-up do you anticipate?

Equipment & supplies notepad and pencil/pen are

minimum requirement, but usually sufficient

video or audio tape optional pros and cons …

Prework / Homework ‘depth’ questions you’d like

participants to answer before or after the interview

not always necessary, but can give broader perspective of the workplace, goals, attitudes, etc.

NOTE: This applies to all field studies, not just CI.

26ISE 412 - 2

To discuss … What are you investigating? (e.g., what types of

market/customers?, doing what?, etc.) What kind of questions? Do you want to “shadow” one or more individuals or

observe a random sample? if individuals, do you want them to do any kind of “pre-work”?

Are you going to try to video or audiotape? (If so, you’ll need permission!)

REMEMBER! … This is your baseline research, so the broader and more

inclusive you can be at this point, the better you will be able to narrow the scope of your project later.

27ISE 412 - 2

Recruiting for a Study Number

depends on the process you’re studying, the target population, etc. typically 10-20, but can be less for smaller, more focused design

problems recruit at the high end of your target sample size to cover ‘drop-

outs’

Diverse, representative sample, considering (as appropriate for the process you are supporting): age range male/female mix type of work, workplace, etc. Ethnic heritage

Once removed from interviewer (friend of a friend) or more familiarity breeds too much shared understanding

28ISE 412 - 2

Interview Techniques: Getting Started (5-10 min.)

DO:Come to the interview prepared:

video equipment ready to gonotes, pen, other materials accessibleright hand free to shake hands at the

door!Set the participant at ease:

introduce yourselves and remind them why you’re there

use equipment setup time to “break the ice”

Select initial interview sitein proximity to “work location” or

starting point, if possibleroom for participant and interviewer to

sit comfortablyask participant where they are most

comfortableMaintain a polite, professional demeanor“set the tone” for the next 2 hours

DON’T: Show up at the door with video

“rolling” Fall into the “buddy trap”

get distracted by side conversations

ask questions about personal matters not related to the study

tell “your” story instead of hearing his/hers

Presume or demand

29ISE 412 - 2

DON’T:Lose sight of the purpose and direction of the interviewby allowing broad motivational discussions to

veer into side issues“gently but firmly” guide the conversation

back on courseDelve into details of what and how

these should be addressed in the contextual interview

Ask leading questions“Suggest an answer” or finish the sentence for them

Suggest judgement of the goals, motivations, activities, etc. (either of the participant or anyone else.)

Interview Techniques: The Initial Interview (15 min.)

DO:Think “broad part of the funnel”

start with broad understanding of goals, motives, priorities, etc.

get “big picture” understanding of their viewpoint

ask for opinions about “tools” and equipment typically used

Listen for and note things you will want to observe and ask about during the contextual interview

Ask open-ended questionsFollow up on interesting points, but maintain pace and direction of the interview

At the end of this phase, move the interview to the “workspace”

30ISE 412 - 2

DON’T:Interrupt the flow unnecessarily.

Ask yourself, “Can this question wait until she/he’s finished with this?”

On the other hand, some questions HAVE to be asked in context - it’s a judgement call!

Ask leading questions.“Is the reason you’re doing that because

___?”“Many people ___. Do you feel the same

way?”“When you say ‘sometimes’, you mean about

once a month, right?”

Interview Techniques: Contextual Interview (60-90 min.)

DO:Watch and listen more than talk and

ask.Provide occasional feedback (“okay,” “I

see”, etc.) to let the participant know you’re listening

Use questions sparingly to gain clarification, understand “why” something is done the way it is, or to guide the participant’s comments

Questions, when warranted, should be open-ended and clear

“I noticed you ___ - why is/was that?”“ What are your thoughts (or feelings)

about that?”“You said you sometimes ____ - when

would you do that and why?”“What do you mean by ‘___’?” “Say

some more about that.”“Tell me what that is for.” “Tell me what

you’re doing now.” (sparingly)

31ISE 412 - 2

Interview Techniques: Contextual Interview (cont.)

DON’T: Finish the sentence or suggest an

answer It is not necessary to fill all silences

Imply judgement in questions or responses “You don’t do THAT, do you?” “Aren’t you afraid of ___?” “Don’t you hate it when ___?”

DO: Take notes, make sketches, write down

questions you want to delve into in detail at the end note what is happening, your

interpretation, your immediate thoughts, etc.

ALWAYS keep your attention on the participant and her/his activities

Pay close attention to key artifacts in the environment: videotape or photograph individual and

groups of items if there are “disposable” items, ask if you

can take them with you

32ISE 412 - 2

Interview Techniques: The Wrap-Up (15 min.)

DO: Ask any remaining questions you

have about the goals, activities, methods, tools, etc.

Summarize key points you heard during the interview, giving the participant a chance to respond, elaborate, correct your interpretation, etc.

Thank the participant for taking the time and allowing you to learn about his/her activities.

Be sure to take all notes, supplies, equipment, etc., when you leave.

DON’T: Overstay your welcome.

Leave any trash behind when you leave.

33ISE 412 - 2

Your turn …

Write one or two questions you will want to ask during your interview.

Test your questions with me as your “participant.” Does the participant understand the question? Does the answer help you to better understand the

situation you are investigating? What follow-up or depth questions are suggested by

the participant’s answer? Revise your questions accordingly. Use this method to help you refine your

interview script.