observation watch, listen, and learn…. agenda observation exercise come back at 3:40. questions?...
Post on 19-Dec-2015
222 views
TRANSCRIPT
Observation
Watch, listen, and learn…
Agenda
Observation exercise Come back at 3:40.
Questions? Observation
Observing Users
Not as easy as you think
One of the best ways to gather feedback about your interface
Watch, listen and learn as a person interacts with your system
Qualitative & quantitative, end users, experimental or naturalistic
Observation
Direct In same room Can be intrusive Users aware of
your presence Only see it one
time May use 1-way
mirror to reduce intrusiveness
IndirectVideo or app recordingReduces intrusiveness, but doesn’t eliminate itCameras focused on screen, face & keyboardGives archival record, but can spend a lot of time reviewing it
Location
Observations may be In lab - Maybe a specially built usability
lab Easier to control Can have user complete set of tasks
In field Watch their everyday actions More realistic Harder to control other factors
UsabilityLab
http://www.surgeworks.com/services/observation_room2.htm
Large viewing area in this one-way mirror which includes an angled sheet of glass the improves light capture and prevents sound transmission between rooms.
Doors for participant room and observation rooms are located such that participants are unaware of observers movements in and out of the observation room.
ObservationRoom
State-of-the-art observation room equipped with three monitors to view participant, participant's monitor, and composite picture in picture.
One-way mirror plus angled glass captures light and isolates sound between rooms.
Comfortable and spacious for three people, but room enough for six seated observers.
Digital mixer for unlimited mixing of input images and recording to VHS, SVHS, or MiniDV recorders.
Task Selection
What tasks are people performing? Representative and realistic? Tasks dealing with specific parts of the
interface you want to test? Problematic tasks?
Don’t forget to pilot your entire evaluation!!
Engaging Users in Evaluation
What’s going on in the user’s head? Use verbal protocol where users describe
their thoughts
Qualitative techniques Think-aloud - can be very helpful Post-hoc verbal protocol - review video Critical incident logging - positive & negative Structured interviews - good questions
“What did you like best/least?” “How would you change..?”
Think Aloud
User describes verbally what s/he is thinking and doing What they believe is happening Why they take an action What they are trying to do
Widely used, popular protocol Potential problems:
Can be awkward for participant Thinking aloud can modify way user performs
task
Cooperative approach
Another technique: Co-discovery learning (Constructive iteration) Join pairs of participants to work together Use think aloud Perhaps have one person be semi-expert
(coach) and one be novice More natural (like conversation) so removes
some awkwardness of individual think aloud Variant: let coach be from design team
(cooperative evaluation)
Alternative
What if thinking aloud during session will be too disruptive?
Can use post-event protocol User performs session, then watches
video afterwards and describes what s/he was thinking
Sometimes difficult to recall Opens up door of interpretation
What if a user gets stuck?
Decide ahead of time what you will do. Offer assistance or not? What kind of
assistance?
You can ask (in cooperative evaluation) “What are you trying to do..?” “What made you think..?” “How would you like to perform..?” “What would make this easier to accomplish..?” Maybe offer hints This is why cooperative approaches are used
Inputs / Outcomes
Need operational prototype could use Wizard of Oz simulation
What you get out “process” or “how-to” information Errors, problems with the interface compare user’s (verbalized) mental
model to designer’s intended model
Capturing a Session
1. Paper & pencil Can be slow May miss things Is definitely cheap and easy
Time 10:00 10:03 10:08 10:22
Task 1 Task 2 Task 3 …
Se
Se
Capturing a Session
2. Recording (audio and/or video) Good for think-aloud Hard to tie to interface Multiple cameras may be needed Good, rich record of session Can be intrusive Can be painful to transcribe and
analyze
Capturing a Session
3. Software logging Modify software to log user actions Can give time-stamped key press or
mouse event Two problems:
Too low-level, want higher level events Massive amount of data, need analysis
tools
Example logs2303761098721869683|hrichter|1098722080134|MV|START|5662303761098721869683|hrichter|1098722122205|MV|QUESTION|false|false|false|false|false|false| 2303761098721869683|hrichter|1098724978982|MV|TAB|AGENDA2303761098721869683|hrichter|1098724981146|MV|TAB|PRESENTATION2303761098721869683|hrichter|1098724985161|MV|SLIDECHANGE|52303761098721869683|hrichter|1098724986904|MV|SEEK|PRESENTATION-A|566|604189|02303761098721869683|hrichter|1098724996257|MV|SEEK|PRESENTATION-A|566|604189|6041892303761098721869683|hrichter|1098724998791|MV|SEEK|PRESENTATION-A|566|604189|6041892303761098721869683|hrichter|1098725002506|MV|TAB|AGENDA2303761098721869683|hrichter|1098725003848|MV|SEEK|AGENDA|566|149613|6041892303761098721869683|hrichter|1098725005981|MV|TAB|PRESENTATION2303761098721869683|hrichter|1098725007133|MV|SLIDECHANGE|32303761098721869683|hrichter|1098725009326|MV|SEEK|PRESENTATION|566|315796|1496132303761098721869683|hrichter|1098725011569|MV|PLAY|566|3157962303761098721869683|hrichter|1098725039850|MV|TAB|AV2303761098721869683|hrichter|1098725054241|MV|TAB|PRESENTATION2303761098721869683|hrichter|1098725056053|MV|SLIDECHANGE|22303761098721869683|hrichter|1098725057365|MV|SEEK|PRESENTATION|566|271191|3157962303761098721869683|hrichter|1098725064986|MV|TAB|AV2303761098721869683|hrichter|1098725083373|MV|TAB|PRESENTATION2303761098721869683|hrichter|1098725084534|MV|TAB|AGENDA2303761098721869683|hrichter|1098725085255|MV|TAB|PRESENTATION2303761098721869683|hrichter|1098725088690|MV|TAB|AV2303761098721869683|hrichter|1098725130500|MV|TAB|AGENDA2303761098721869683|hrichter|1098725139643|MV|TAB|AV2303761098721869683|hrichter|1098726430039|MV|STOP|566|2711912303761098721869683|hrichter|1098726432482|MV|END
Analysis
Many approaches Task based
How do users approach the problem What problems do users have Need not be exhaustive, look for interesting
cases Performance based
Frequency and timing of actions, errors, task completion, etc.
Very time consuming!!
Example: Heather’s study
Software: MeetingViewer interface fully functional
Criteria – learnability, efficiency, see what aspects of interface get used, what might be missing
Resources – subjects were students in a research group, just me as evaluator, plenty of time
Wanted completely authentic experience
Heather’s evaluation
Task: answer questions from a recorded meeting, use my software as desired
Think-aloud Video taped, software logs Also had post questionnaire Wrote my own code for log analysis Watched video and matched behavior to
software logs
Example: Analysis
Printed logs Watched video and marked actions,
thoughts, and video timing info on the logs
Wrote down interesting quotes from the think aloud
Took 2 to 3 times the session time just to do this analysis alone!