Dr Jon Dodd

BEHAVIOURAL RESEARCH BLOOPERS

June 20th 2019

& HOW TO AVOID THEM

2

1999-2019 CEO Bunnyfoot

Hi - DR JON DODD

1990-1999 Neuroscience researcher (and Guinee pig)

POSITIVE, PROFITABLE & REWARDING

Since 1999 a leading specialist UX consultancy

WE HELP OUR CLIENTS CREATE

Experts in:

+ Providers of UX related services:

UX STRATEGY

SERVICE DESIGN

SERVICE INNOVATION

USER/CUSTOMER RESEARCH

USER CENTRED DESIGN

DESIGN THINKING

USABILITY TESTING

PERSUASION DESIGN

EYE TRACKING

ACCESSIBILITY

USER EXPERIENCES

Some of the clients we are currently

working with (May/June 2019)

A 40 strong team of psychologists, researchers, experience designers, usability & customer experience specialists

based out of London, Oxford, and Sheffield

▪ LEADING UX TRAINING PROVIDER

▪ LAB HIRE & LAB BUILD SERVICE

▪ FIELDWORK RECRUITMENT

4

LTM & FORGETTING

Learning requires rehearsal or conscious repetition

Hermann Ebbinghaus 1886

5

RESEARCH IS INHERENT TO GOOD DESIGN

e.g. in User Centred Design

6

IN USER CENTRED DESIGN

learn about your

Business

Business requirements/ stakeholder workshop

Participatory design/concept testing

Strategy and goal workshops

Requirement analysis planning

Business to customer uncovery

Desk research

learn about your

Competitors

Customer research/ benchmarking

Best practice analysis/feature analysis

learn about your

Customers

Behaviour observation

▪ Observed browsing

▪ Customer Inquiry

▪ Ethnography

▪ Competitor testing

Diary studies

Audience surveys

Depth interviews

User/Customer requirement workshop (focus groups)

define the

Information

architecture

Information architecture development

Content strategy

develop the

Interaction design

Paper prototyping

Iterative wireframe development

Clickable prototype development

document with

Blueprints /instructions

Information architecture blueprints

Interaction design documentation

▪ Wireframes

▪ Templates

▪ Elements

long term

Support ‘Rent-a-guru’ ongoing ‘’on

call’ immediate access to our experts

Workshop, training and seminars

Policy documentation

regular

Monitoring Advanced visitor analysis

Benchmark quantitative testing

Competitor monitoring/trend spotting

Social media monitoring

Satisfaction surveys and interviews

ongoing

Optimisation A/B testing

Multi-variate testing

Updated customer intelligence

Design Modification updates

Persuasion auditing

define

Customer Models

Persona development

Scenario and use case setting

explore

Customer interaction

Content audit and analysis

Ideation/concepting

Participatory design workshop

Mental modeling

Mental model/content model

develop

Testable concepts

Concept rationalisation

Information architecture concept generation

Storyboarding

Lo-fi prototype sketching

research

Creative concepts Creative research

exploration

Creative workshop

evolve and

Develop the creative Creative concept

generation

Creative application

▪ Templates

▪ Elements

develop

Design guides/assets Design guide production

Design asset production

▪ Layered psd, png, etc,

▪ Presentation layer code

RESEARCH MODEL ARCHITECT DESIGN OPTIMISE

Expert usability eval

User testing

Accessibility audit

Prototype testing

Eye tracking

Card sorting

Navigation testing

Look and feel testing

Remote testing

Analytics

…and more tailor made for you

TESTING AND EVALUATION

7

IN SERVICE DESIGN

Management expectations

Customer

expectationsKnowledge gap Do your research to reduce this

gap

Management expectations

Servicespecification

Try to set spec to meet customer

expectations

Policy gap

Service Specification

Servicedelivery

Attempt to deliver the spec

(resources etc.)

Delivery gap

Service delivery

Customer coms

Communication realistically and

truthfully

– help set expectations

Communication gap

Note: this is a model we use called the gap model to help frame service design

Customer

perception

Overall goal:

Reduce this gap!

Research is essential to reduce the knowledge gap this is the difference between what the org thinks its users/customers want and need – and what they actually need.

8

4D OR DOUBLE DIAMOND

9

DESIGN THINKING…

10

11

DESIGN SPRINTS

Define the problem & start the UX

Canvas

Learn from others &

develop ideas

Rapidly evolve ideas & produce

visualisation(s)

Build suitable prototype(s)

Test & determine next steps

Monday Tuesday Wednesday Thursday Friday

12

ANYONE RECOGNISE THIS?

13

THE BIG PICTURE

Kruger & Dunning 1999

…people who lack the knowledge or wisdom to perform well are often unaware of this fact.

We attribute this lack of awareness to a deficit in metacognitive skill. That is, the same incompetence that leads them to make wrong choices also deprives them of the savvy necessary to recognize competence, be it their own or anyone else’s.

14

PERHAPS THE POSTER CHILD OF THE MOMENT…

15

16

A NICE ONE

17

…EASIER TO REMEMBER AND MORE POINTED

18

AND HAS BEEN APPLIED TO MANY INDUSTRIES

19

IT’S ACTUALLY A BIT MORE COMPLICATED THAN THAT…

…which is a bit ironic…

20

BUT A USEFUL TOOL…

Essentially all models are wrong – but some are useful

George E P Box

21

ASK YOURSELF REGULARLY – WHERE ARE YOU?

UX and research etc.

I hope I am here – every

day a school day

I did stand-up comedy

I’m about to

do my

second ultra

marathon

22

This proves this

RESEARCH: SOME THINGS I HEAR REGULARLY

We have loads of

[survey] data

ask them this…

We need more

people (a bigger

sample)

I want [eyetracking, focus group,

interviews…]

You’re an expert –

why do we need

research

That wasn’t a good test…

(moderator didn’t say enough)

(a child kept running in)

But we know our

customers

Just ask them what

they think about

this

23

…AND IN DESIGN (PARTICULARLY INTERACTION & SERVICE DESIGN)

24

RESEARCH: WHAT WE WOULD LIKE TO SEE MORE OF

What’s the best

approach(s)…

These are our

research

objectives

How can we

improve…

This is our problem

Every day is a

school day

25

You are not your audience/user/customer *

A FUNDAMENTAL TRUTH

You do not…

See things like they do

Know what they know

Want what they want

Work how they work

This is critical information when designing a product or service

…and btw your bosses have even less clue than you do…

* delete as appropriate!

…that you will often have to remind yourself and your colleagues about

26

27

MORE FROM WHAT KIDS CAN TEACH US…

28

ANYONE KNOW THIS CHAP?

Don’t Make Me Think (Revisited), 2014

It’s only natural to assume that everyone uses the Web the same way we do, and - like everyone else – we tend to think that our own behaviour is much more orderly and sensible than it really is

Steve Krug, 2014

29

WORTH READING…. AND GIVING TO YOUR BOSS?

30

ROLF MOLICH’S COMMON USABILITY EVALUATION (CUE) SERIES

Look it up – it might scare you

The latest one was about skilful

moderation

Participants were some leading lights

from usability evaluation (e.g.

president of UXPA, writers of popular

usability and usability testing books…)

and others

A wide variety of ‘mistakes’ and less

than best practice practice observed

Every day a school day

31

JUNK IN – JUNK OUT

It is essential to choose the right methodology

….good reminder – or a story you can tell those you might need to

!

32

THE STORY OF THE HEALTHY HAMBURGER?

33

THE STORY OF THE HEALTHY HAMBURGER!

Q?So what would

be the correct

approach?

34

ANOTHER EXAMPLE

If I asked you what you looked

at – would you lie?

Be truthful!

35

YOUR CHALLENGE

Which is from 30 Men?

Which is from 30 women?

36

ITS JUST PAST MIDDAY...

37

FILL IN THE BLANKS

38

ULTIMATELY...

People don’t remember...

People lie (confabulate)...

People post rationalise...

People hate to appear stupid...

People are bad at introspection...

People will be kind to you and tell you what they think you want to

hear

39

and…most of our behaviour including ‘complex’ behaviour is non-conscious (sub-conscious)- So we need to design (and design our research) with this in mind

40

EXERCISE: WHICH METHODS GO WHERE…

Shout some out…

41

HERE IS ONE THEY PREPARED EARLIER…

42

LETS LEARN FROM SOME BLOOPERS & HORROR STORIESEvery day is a school day…

43

RESEARCHING THE WRONG PEOPLE…

Under-specification

Over-specification

Bias

What about the ‘unknown unknowns’?

44

RESEARCHING THE WRONG PEOPLE…

‘Professional’ research participants

Fakes

Dunning Kruger sufferers (usage, expertise)

The lovelorn

(the smelly)

45

RESEARCHING THE RIGHT PEOPLE…

Take recruitment very seriously

Test screeners and modify appropriately

Know the flex points

Go where they flock

46

FALLING IN LOVE WITH A METHOD

The tyranny of eyetracking

The ease of the survey deployment

The fame of the focus group

47

= BE PROMISCUOUS WITH METHODS

Start with the research objectives

Use a mix of methods

Be innovative

48

TALKING TOO MUCH

Asking too many questions

Interrupting flow

(leading Qs, closed Qs etc.)

49

= WATCH, LISTEN & LEARN

Think very carefully before speaking

Power of silence

Practice, checklists etc.

50

ROBOTIC ADHERENCE TO A SCRIPT

Soulless Q & A

No empathy

No Engagement

No discovery

51

= BE FLEXIBLE, BE INTERESTED

Be prepared (and confident) to explore

Be interested (and look like you are)

Have empathy

Respond appropriately

52

THE WRONG NUMBER

Often misinterpreted

Too many = wasted ROI = too much to

analyse (paralysis through analysis)

Scientific significance v trend v

confidence

53

THE RIGHT NUMBER…

Rules of thumb for qual

Calculators for quant

Repeats

54

AND LOTS MORE TO WATCH OUT FOR

Confirmation bias (selective nature for research particularly important)

Expectation (experimentors/observer) bias (along with confirmation bias dangerous for the truth)

Availability heuristic (take notes, independent observers, group analysis)

Courtesy bias (look out for people doing this – occurs a lot in focus groups, at the end of user testing)

Curse of knowledge (especially important in research – domain experts lack empathy for users)

Group think (focus groups especially)

Interoceptive bias (well fed etc. changes outlook of researcher/judge)

Mere exposure effect (familiarity breeds… in this case positivity – important in ratings etc. e.g. satisfaction)

False consensus effect (you are not your user… and neither is your husband, wife, mother etc…)

55

CONCLUSIONS

Assess where you are on the curve

Every day is a school day

Observe behaviour where possible

Use mix methods and be flexible

Avoid and learn from bloopers

56

FOLLOW US

LONDONT: 0207 608 1670

5th Floor,

54 St John's Square,

Farringdon, London,

EC1V 4JL

OXFORDT: 01235 838 514

Harwell Innovation Centre,

173 Curie Avenue

Harwell, Oxfordshire, OX11 0QG

SHEFFIELDT: 0114 478 2950

The Old Chapel

8 Mortimer Street

Sheffield, S1 4SF

THANK YOU!

Questions…?