seeing is believing: using systematic observations in user experience research. march 11 th, 2011...

Seeing is believing:

Using systematic observations in user experience research.

March 11th, 2011

Bram van Mil b.van.mil@noldus.nlSales consultant

Noldus Information Technology

Systematic Observations

Measuring Behavior with The Observer XT

Observation labs and other research tools

Agenda

International developer of professional software for behavioral research

From a single software license up to a complete observation lab

Complete with installation, training and support

HQ located in Wageningen, The Netherlands

Founded in 1989, currently ≈

115 employees

Customers in 75 countries

Installed base Observer

software ≈ 4400 organizations

Who we areNoldus Information Technology, since 1989…

Why study behavior?

Develop new drugs to cure diseases

Support medical diagnosis and therapy

Support human resource management

Improve educational systems

Improve animal welfare

Control insect pests

Create more usable products

Create safer work environments

Optimize team performance

Etc.

Questionnaires

Observations

Ways to study behavior

• Problem of meaning (interpretation of a question)

• Problem of omission (inadvertently leaving out our relevant information)

• Problem of memory

• Social desirability effect

• Question threat (questions may appear threatening)

• Interviewer characteristics (influence on participant)

• Gap between stated and actual behavior

Disadvantages of questionnaires

Record who does what, where, when and with whom

The alternative: Systematic observations

Objective

Accurate

Standardized

Quantify qualitative data

Advantages of systematic observations

A human observer watches the behavior of thesubjects and writes down the observations on paper

and uses a clock for time-information.

Disadvantages: Subjective measurements

When writing, the observer cannot observe

Very labor intensive

No integration with other modalities

Measuring Behaviorthe pioneers way…

Create a coding scheme before or during the observations and annotate behavior by a simple key press.

This will result in an event-log which contains time-stamped events that can be synchronized with multimodal data:

Videos

Physiological data

Eye tracking data

Etc.

Solution: The Observer XTfor structured observations

How to execute systematic observations

Three steps

1. Setup

2. Observe

3. Analyze

How to execute systematic observations

Step 1 – SetupObservation sources

How to execute systematic observations

Step 1 – SetupCoding scheme

Subjects

Behaviors

- Point events

- State events

Modifiers

How to execute systematic observations

Step 1 – SetupCoding scheme : Example

Time Subject, behavior, modifiers

09:35:12 Joe, run, fast, North

09:40:23 Mary, talk, Rose

09:40:24 Rose, smile

09:40:24 Rose, affect, 5

09:40:31 Mary, smile

How to execute systematic observations

Step 1 – SetupCoding scheme : Mutually exclusive behaviors

Mutually exclusive: one behavior ends the previous

How to execute systematic observations

Step 2 – Observe

Live scoring:

With or without videos

Offline scoring:

From digital video files

How to execute systematic observations

Step 3 – Analyze Data selection

Choose the data you want to analyze by filtering and nesting your data

This is done by data profiles:

In order to save different criteria for you data selection and data analysis

How to execute systematic observations

Step 3 – Analyze Visualization of your data

How to execute systematic observations

Step 3 – Analyze Descriptive statistics

Advanced analysis - Lag-sequential analyses - Reliability analysis

Statistical analysis - Export to Excel, SPSS or as ASCI file

Export highlight videos

How to execute systematic observations

Step 3 – Analyze: Numerical analysis on external data

Other tools

That can be used in behavioral research

Neutral

Happy

Sad

Angry

Surprised

Scare

Disgusted

Also: gender, age, ethnicity, glasses and facial hair

FaceReaderDetect facial expressions

Eye Tracking SystemsSee what your participants are looking at

Computer monitor / Standalone / Head mounted

Gaze tracks, heat maps, fixations and areas of interest

Physiological dataDetect mental load, physical load and emotional states

ECG – Electro Cardio Gram

EMG – Electro Myo Gram

EEG – Electroencephalogram

GSR – Galvanic Skin Response

Labs

Complete setups for behavioral research

Portable Usability Lab

Portable Observation Lab

Custom Mobile Observation Labs

Stationary Observation Lab

Game Experience LabGhent University

Two room Usability Lab for User Experience research in gaming on consoles and pc’s

Game Experience LabGhent University

Game Experience LabGhent University

Different facilities under 1 roof

– Observational-, sensory- and mood- labs

Observe eating behavior

– Introduce better suited products

Restaurant of the FutureWageningen University

Restaurant of the FutureWageningen University

Service, training and supportWe are there when you need us

On-site installation and training Behavioral scientists

Hardware engineers

Excellent technical support Helpdesk

Online knowledge base

Other Service contracts

Measuring Behavior conference

Bram van Mil b.van.mil@noldus.nlSales consultant