in4matx 231 human-computer interaction presenters: ritesh subramanian tanmay goel observing users
TRANSCRIPT
IN4MATX 231 IN4MATX 231 Human-Computer Human-Computer
InteractionInteraction
Presenters:Presenters:
Ritesh Ritesh SubramanianSubramanian
Tanmay GoelTanmay Goel
Observing UsersObserving Users
OutlineOutline
Benefits and challenges of different Benefits and challenges of different types of observations.types of observations.
How to observe as an onlooker, a How to observe as an onlooker, a participant, and an ethnographer.participant, and an ethnographer.
How to collect, analyze and present How to collect, analyze and present observational data.observational data.
Indirect observation using diary Indirect observation using diary studies and logging.studies and logging.
Qualitative and Quantitative analysis.Qualitative and Quantitative analysis.
What and when to observeWhat and when to observe
Observation is usable at all stages during Observation is usable at all stages during product development.product development.
Goals and questions determine the Goals and questions determine the paradigms and techniques used.paradigms and techniques used.
Observers can be:Observers can be:– OnlookersOnlookers– ParticipantsParticipants– EthnographersEthnographers
The degree of immersion that evaluators The degree of immersion that evaluators adopt varies across a broad outsider-adopt varies across a broad outsider-insider spectrum.insider spectrum.
Types of ObservationTypes of Observation
ObservationObservationControlled Controlled
Environment Environment (e.g. Lab)(e.g. Lab)
Field Field Environment Environment (e.g. Natural)(e.g. Natural)
Outsider looking on
““Quick and Dirty”Quick and Dirty”
In usability In usability testingtesting
““Quick and Dirty”Quick and Dirty”
In field testingIn field testing
InsiderInsider N.AN.A..
Participant Participant observation (e.g. observation (e.g. in ethnography)in ethnography)
Approaches to ObservationApproaches to Observation
““Quick and Dirty” observationQuick and Dirty” observation– It can occur anywhere, anytimeIt can occur anywhere, anytime– Least formality involvedLeast formality involved
Observation in usability testingObservation in usability testing– Video and interaction logs capture all the Video and interaction logs capture all the
user operationsuser operations– One-way mirrors or remote TV screenOne-way mirrors or remote TV screen– Data is used to analyze what users are Data is used to analyze what users are
doing and provide insight into users’ doing and provide insight into users’ affective reactionsaffective reactions
Approaches to ObservationApproaches to Observation
Observation in field studiesObservation in field studies– The observer may be a passive observer, participant The observer may be a passive observer, participant
observer or an ethnographer.observer or an ethnographer.– Goal is to cause as little disruption as possible.Goal is to cause as little disruption as possible.– Passive or outside observer – An observer in a class of Passive or outside observer – An observer in a class of
boys and girls whose primary job to keep track of how boys and girls whose primary job to keep track of how much time a particular technology is used by each much time a particular technology is used by each gender.gender.
– Participant observer – Participates in social conventions Participant observer – Participates in social conventions of a group, combines participation with maintenance of of a group, combines participation with maintenance of professional distance for unbiased observation.professional distance for unbiased observation.
– Ethnography – Takes weeks, months or longer. Inside Ethnography – Takes weeks, months or longer. Inside information of community activity is obtained.information of community activity is obtained.
How to observe:How to observe:In controlled environmentIn controlled environment
Practical issues that need to thought of in advance:Practical issues that need to thought of in advance:– Decide where users are located so that equipment Decide where users are located so that equipment
can be setup in advancecan be setup in advance– Decide about modes of data capture e.g. video, Decide about modes of data capture e.g. video,
interaction logsinteraction logs– Equipment testing to get expected performance Equipment testing to get expected performance
(audio and video)(audio and video)– Get legal user consentGet legal user consent– Prepare a script to guide users through the set of Prepare a script to guide users through the set of
questions questions – It is important to make users feel comfortableIt is important to make users feel comfortable– Problem with this approach: Observers do not know Problem with this approach: Observers do not know
what users are thinking.what users are thinking.
Controlled Environment – Controlled Environment – Think-Aloud TechniqueThink-Aloud Technique
The user is trying to perform a certain The user is trying to perform a certain task. Observer wonders, what is going task. Observer wonders, what is going on, what is he thinking, why did he do on, what is he thinking, why did he do that, etc.that, etc.
Eternalize thought process.Eternalize thought process. A big problem is occurrences of silence A big problem is occurrences of silence
during think-aloud process.during think-aloud process. Maybe have two people work together Maybe have two people work together
and talk to each other.and talk to each other.
Controlled Environment –Controlled Environment –Think Aloud TechniqueThink Aloud Technique
Checklist of things to plan Checklist of things to plan before going into the fieldbefore going into the field
State the initial goal and questions clearlyState the initial goal and questions clearly Select a framework to guide activities in the fieldSelect a framework to guide activities in the field Decide how to record eventsDecide how to record events Think about how to gain acceptance and trust of users Think about how to gain acceptance and trust of users
under observationunder observation Be prepared to go through these notes to weed out Be prepared to go through these notes to weed out
ambiguitiesambiguities Highlight and separate personal opinion from what Highlight and separate personal opinion from what
happenshappens Be prepared to refine and refocus the study for same or Be prepared to refine and refocus the study for same or
new user groupsnew user groups Think about how to handle sensitive issuesThink about how to handle sensitive issues Consider working as a team; conform notes with another Consider working as a team; conform notes with another
membermember Consider checking notes with an informant and plan to Consider checking notes with an informant and plan to
look at the situation from a different perspectivelook at the situation from a different perspective
How to observe (contd.):How to observe (contd.):In the fieldIn the field
Frameworks to guide observationFrameworks to guide observation– The personThe person. Who? . Who? – The place.The place. Where? Where?– The thing.The thing. What? What?
The Goetz and LeCompte (1984) framework:The Goetz and LeCompte (1984) framework:– WhoWho is present? is present? – WhatWhat is happening? is happening? – WhenWhen does the activity occur? does the activity occur?– WhereWhere is it happening? is it happening? – WhyWhy is it happening? is it happening? – HowHow is the activity organized? is the activity organized?
In the fieldIn the field
The Robinson (1993) frameworkThe Robinson (1993) framework– SpaceSpace. What is the physical space like?. What is the physical space like?
– ActorsActors. Who is involved?. Who is involved?
– ActivitiesActivities. What are they doing?. What are they doing?
– ObjectsObjects. What objects are present? . What objects are present?
– ActsActs. What are individuals doing?. What are individuals doing?
– EventsEvents. What kind of event is it?. What kind of event is it?
– GoalsGoals. What do they to accomplish?. What do they to accomplish?
– FeelingsFeelings. What is the mood of the group and of . What is the mood of the group and of individuals?individuals?
How to observe (contd.):How to observe (contd.):Participant observer or ethnographyParticipant observer or ethnography
Checklist for doing ethnography Checklist for doing ethnography – Identify the problem or goal and ask good Identify the problem or goal and ask good
questions to be answered by the studyquestions to be answered by the study– The most important part of fieldwork is being there The most important part of fieldwork is being there
to observe, ask questions and record what is seen to observe, ask questions and record what is seen and heardand heard
– Collect variety of data. E.g. Notes, still pictures, Collect variety of data. E.g. Notes, still pictures, audio and videoaudio and video
– Be prepared to move backwards and forwards Be prepared to move backwards and forwards between the broad picture and specific questions between the broad picture and specific questions
– Analyze the data using a holistic approach in which Analyze the data using a holistic approach in which observations are contextualizedobservations are contextualized
Data Collection TechniquesData Collection Techniques
Notes plus still cameraNotes plus still camera– The least technical way of collecting dataThe least technical way of collecting data– Transcription the first step in data analysisTranscription the first step in data analysis– Photographs, sketches, etc. can be easily collectedPhotographs, sketches, etc. can be easily collected
Audio recording plus still cameraAudio recording plus still camera– Less intrusive than video Less intrusive than video – More flexible and mobile More flexible and mobile – One drawback is transcribing the data which can One drawback is transcribing the data which can
be onerous if many hours have to be transcribedbe onerous if many hours have to be transcribed
Data Collection TechniquesData Collection Techniques
VideoVideo– Advantage of both audio and Advantage of both audio and
video data; but more intrusivevideo data; but more intrusive– Attention becomes focused on Attention becomes focused on
what is seen through the lens what is seen through the lens – Analysis of video can be long Analysis of video can be long
and time consumingand time consuming– Sound may get muffled when Sound may get muffled when
recording in noisy conditionsrecording in noisy conditions
ComparisonComparison
Table source: Preece, Rogers, Sharp: Interaction Design: Beyond Human-Computer Interaction. Wiley
Comparison (contd.)Comparison (contd.)
Table source: Preece, Rogers, Sharp: Interaction Design: Beyond Human-Computer Interaction. Wiley
Indirect observation: Indirect observation: tracking user activitiestracking user activities
DiariesDiaries– Diaries provide a record of what users did, when Diaries provide a record of what users did, when
they did it and what they thought about their they did it and what they thought about their interactions with technologyinteractions with technology
– Useful when users and scattered and unreachableUseful when users and scattered and unreachable– Inexpensive and require no special equipment or Inexpensive and require no special equipment or
expertiseexpertise– They rely on participants being reliable and They rely on participants being reliable and
remembering to complete themremembering to complete them– Participants usually remember incidents for better Participants usually remember incidents for better
or for worseor for worse
Indirect observation: Indirect observation: tracking user activitiestracking user activities
Interaction loggingInteraction logging– Includes analysing data from key presses, mouse Includes analysing data from key presses, mouse
movements etc.movements etc.– Usually synchronized with video and audio logsUsually synchronized with video and audio logs– Time stamped to calculate how much time a user Time stamped to calculate how much time a user
spends on a particular taskspends on a particular task– E.g. Explicit counters that record visits to a E.g. Explicit counters that record visits to a
website. If you want to find out if adding a bulletin website. If you want to find out if adding a bulletin board increases the number of visits, comparison board increases the number of visits, comparison of traffic before and after is usefulof traffic before and after is useful
– Unobtrusive and large volumes of data can be Unobtrusive and large volumes of data can be collectedcollected
Interaction Logging- Interaction Logging- Webtrends Log Analyzer Webtrends Log Analyzer
Interaction LoggingInteraction Logging
Ethical Concerns: They don't know Ethical Concerns: They don't know we are watching. Shall we tell them?we are watching. Shall we tell them?
If we tell users about logging they may react or If we tell users about logging they may react or change their behaviorchange their behavior
It depends on the context, and how much personal It depends on the context, and how much personal information is collectedinformation is collected
e.g. Companies tell about data logged for quality e.g. Companies tell about data logged for quality assurance purposesassurance purposes
Concerns arise when personal information is used Concerns arise when personal information is used to infer financial or health informationto infer financial or health information
Analyzing, interpreting and Analyzing, interpreting and presenting the datapresenting the data
Three main types of data are exploredThree main types of data are explored QualitativeQualitative datadata - - interpretedinterpreted & used to tell the & used to tell the
‘story’ about what was observed‘story’ about what was observed. . Qualitative dataQualitative data - - categorizedcategorized using techniques using techniques
such as content analysis.such as content analysis. Quantitative dataQuantitative data - collected from interaction & - collected from interaction &
video logs. Presented as values, tables, charts, video logs. Presented as values, tables, charts, graphs and treated statistically.graphs and treated statistically.
Qualitative analysis to tell a Qualitative analysis to tell a storystory
Review the data after each observation to Review the data after each observation to identify key themes identify key themes
Record themes in a coherent yet flexible form Record themes in a coherent yet flexible form – E.g. Laptops, audio recordingsE.g. Laptops, audio recordings
Record the date and time of each data sessionRecord the date and time of each data session As themes emerge, check your understanding As themes emerge, check your understanding
with informants or people you observewith informants or people you observe Iterate until the story reflects observationIterate until the story reflects observation Report findings to a group in the form of a Report findings to a group in the form of a
presentation or written reportpresentation or written report
Analyzing and reporting Analyzing and reporting ethnographic dataethnographic data
Look for key events within a group that drive a Look for key events within a group that drive a group’s activitygroup’s activity
Look for patterns or behavior among different Look for patterns or behavior among different playersplayers
Compare sources of data against each other Compare sources of data against each other Report findings in a convincing and honest wayReport findings in a convincing and honest way e.g. Software tools such as Ethnograph allow e.g. Software tools such as Ethnograph allow
ethnographers to code their notes so that they can be ethnographers to code their notes so that they can be searched, sorted and retrieved. The information can searched, sorted and retrieved. The information can be printed out as a tree showing relationships of be printed out as a tree showing relationships of occurrences. occurrences.
EthnographEthnograph
EthnographEthnograph
Qualitative analysis for Qualitative analysis for categorizationcategorization
Looking for incidents or patternsLooking for incidents or patterns– Look for critical incidents, such as times when the Look for critical incidents, such as times when the
user was obviously stuck. (marked by silence, looks user was obviously stuck. (marked by silence, looks of confusion)of confusion)
– E.g. Jurgen Koenemann-Belliveau et al(1994) used E.g. Jurgen Koenemann-Belliveau et al(1994) used this approach to compare the efficacy of two this approach to compare the efficacy of two versions of a Smalltalk programming manual for versions of a Smalltalk programming manual for novice programmers.novice programmers.
– They were able to identify specific problems by They were able to identify specific problems by tracing through sequence of incidents and thereby tracing through sequence of incidents and thereby achieving a holistic understanding.achieving a holistic understanding.
Qualitative analysis for Qualitative analysis for categorizationcategorization
Analyzing data into categoriesAnalyzing data into categories– Content analysis provides a reliable and systematic way Content analysis provides a reliable and systematic way
of coding content into meaningful sets of mutually of coding content into meaningful sets of mutually exclusive categoriesexclusive categories
– Categories should be orthogonal i.e. no overlapCategories should be orthogonal i.e. no overlap– Appropriate granularity must be selected Appropriate granularity must be selected – e.g. Training two people to use categories and letting e.g. Training two people to use categories and letting
them analyze the same data sample. If huge discrepancy them analyze the same data sample. If huge discrepancy in the analyses is observed, then either the training was in the analyses is observed, then either the training was insufficient or categories need to be redefinedinsufficient or categories need to be redefined
– Inter-research reliability rating- Percentage of agreement Inter-research reliability rating- Percentage of agreement between two researchers, defined as ratio of number of between two researchers, defined as ratio of number of items categorized in the same way to the total number of items categorized in the same way to the total number of items observeditems observed
Qualitative analysis for Qualitative analysis for categorizationcategorization
Analyzing DiscourseAnalyzing Discourse– Focus on the dialog, i.e. meaning of what was said Focus on the dialog, i.e. meaning of what was said
rather than contentrather than content– Discourse analysis is strongly interpretiveDiscourse analysis is strongly interpretive– Language is viewed as a constructive tool and Language is viewed as a constructive tool and
discourse analysis provides a way of focusing upon discourse analysis provides a way of focusing upon people’s use of language to construct versions of people’s use of language to construct versions of their worldstheir worlds
– E.g. Analyzing discourse on the Internet (e.g. in E.g. Analyzing discourse on the Internet (e.g. in chat rooms, bulletin boards) has started to chat rooms, bulletin boards) has started to influence designers’ understanding about user influence designers’ understanding about user needs in these environmentsneeds in these environments
Quantitative data analysis Quantitative data analysis and feedbackand feedback
Quantitative data analysisQuantitative data analysis– Video data collected is usually annotated to calculate Video data collected is usually annotated to calculate
performance times performance times – The data stream from interaction logs is used to calculate The data stream from interaction logs is used to calculate
performance timesperformance times– Data is analyzed using means, standard deviations, t-Data is analyzed using means, standard deviations, t-
tests etc.tests etc. Feeding the findings back into the designFeeding the findings back into the design
– Written reports make for easy reading and a good Written reports make for easy reading and a good reference documentreference document
– Include anecdotes, pictures, quotations and video clips to Include anecdotes, pictures, quotations and video clips to bring study to life and stimulate interestbring study to life and stimulate interest
– Verbal presentations can be a powerful mode of feedbackVerbal presentations can be a powerful mode of feedback
SummarySummary
Observation in usability study tends to be objectiveObservation in usability study tends to be objective In contrast, in participant observation the evaluator works In contrast, in participant observation the evaluator works
with the user to understand their activities and beliefswith the user to understand their activities and beliefs Ethnography uses a technique wherein the ethnographer Ethnography uses a technique wherein the ethnographer
immerse themselves in the culture they studyimmerse themselves in the culture they study Observational data collection depends on the paradigm Observational data collection depends on the paradigm
usedused Combination of video, audio, diaries and logs can be used Combination of video, audio, diaries and logs can be used
to collect observational datato collect observational data Analyzing video and data logs can be tedious due to the Analyzing video and data logs can be tedious due to the
sheer volume of datasheer volume of data Evaluators flag events in real time and return to examine Evaluators flag events in real time and return to examine
them later. them later. Identifying key events is an effective approachIdentifying key events is an effective approach