Models and Methods for Global User Research

Robert Schumacher, PhD.Managing DirectorUser Centric, Inc

Robert M. Schumacher

Managing Director

Robert M. Schumacher

Managing Director

User experience research in many places…

What is this talk not about

• Globalization or localization of test artifacts

“The World is Flat” – Thomas Friedman Pace of technology introduction is increasing Global companies introducing products in

unfamiliar markets and market segments Products / services are getting more complex Internet and mobility change what we work on

and the way we work

Increasing demand for global user

research

Connection of Cultures

Must find ways to do user research better, faster, and (often) cheaper than we do today

Irrelevance of Place

Compression ofTime

• The Internet culture creates expectations – Partners / clients /

collaborators / developers around the globe create work schedules that are on “world time”

• Speed is a feature of user research – Agile development

meets user research must adapt methods

– Example: Six country web site review done in 24 hours

Compression ofTime

• Outsourcing of core and supporting activities– Some kinds of user

research can be outsourced to different time zones to achieve rapid turn around

– Example: Overnight transcription of usability tests

• New tools enable faster production of artifacts– Automated data collection /

summarization tools– Example: Digital video

editing

Compression ofTime

More research, more data…more faster

Remote

Face to face

Methods

New technologies and techniques allow forRemote:

– Moderated testing– Unmoderated testing– Observation

Irrelevance of Place

Remote Moderated TestingProducts like GotoMeeting allow connections to the test (or observation) computer to the Internet. VoIP can carry voice cheaply.

LiveMeetingWebEx

GoToMeetingFor screen VoIP Audio

SkypeGoogleTalk

Translator

ModeratorParticipant

Observers

• Hundreds of users agree to participate in a study• In their natural context• From geographically spread locations• Users try to complete tasks + answer questions• No human moderation needed • Browser bar connects users with secure servers

Find the umbrella.

Remote Unmoderated Testing

Remote Unmoderated Testing

‘Task-based’

Surveys> Online/remote Usability Studies (unmoderated)> Benchmarking (competitive /comparison)> UX Dash`boards (measure ROI)

Online Card Sorting> Open or closed> Stand alone or> Integrated with task-based studies & surveys

Online Surveys> Ad hoc research> Voice of Customer studies> Integrated with Web Analytics data

User Recruitin

g Tool> Intercept real visitors (tab or layer)> Create your own private panel> Use a panel provider*

Robust Set of Services

Remote

Face to face

Methods

• Sometimes ‘place’ matters and it seems like you must test with users F2F

• Some guidelines…

When to do…Remote Testing

• Web/software UI

• More quantitative based

• Large, distributed sample or low incidence

• Low(er) budget• High penetration of

Internet access

In Person• Physical artifact• Need rich qualitative

feedback• Need to have the human

connection• Ensure high-level of

consistency• Uncertain of quality or

environment

Recent Research Questionsthat suggested face to face research was needed

• Why are conversions are lower in Brazil than in the rest of the world?

• What is the customer journey for our product suite in Japan?

• Is our site perceived to be ‘Chinese’ (enough)?• Do my translations into Spanish affect performance?• We have a new navigation system for our HDTV – how

usable is it for American consumers?• Development wants to be certain that the site ‘works’ for

our global users before it is launched…in 18 countries

You cannot and should not…

conduct user research unless you know

what the purpose of the study is.

DUH!However…

80%A lot of the time we spend is NOT on the technical.

It ’s on planning and logistics.

Planning the conditions to collect good data.

Increasing demand for user testing across borders– Usability practices can vary

widely country to country– Many issues to consider for

fieldwork…• Moderators• Interpreters and translations• Recruiting and incentives • Facilities

Connection of Cultures

Impossible to over-prepare

On-site travel

Two Models of F2F Global User Research

“Over the transom”Work remote with local

partners

Images person in easy chair, or airline seat

Travel to destinations?- Cost: Time & money!- Benefit:

- Richer data- Higher consistency- Lower risk of failure

Stay put…work remote? - Cost:

- Worry- Possible quality issues

- Benefit:- Lower cost- Faster turn around

Four Elements of User Research Projects

Project Management

Analysis & Reporting

Fieldwork

Preparation

Preparation: Recruiting• Recruiting practices differ widely

– Privacy: Lists…sigh

So you need to test in…

Be prepared for (very) high recruitment costs

Recruiting• Recruiting practices differ widely

– Privacy: Sometimes a list is bad, really bad– Selecting participants…or not

• Incidence and participation differs– How many “extras” or “floaters” do you

need? What is the “show” rate?– What did you pay your participants?

Vive La France!

So you need to test in…

Find creative ways to incent participants

32

Interpreters• “Think aloud” data is simply more difficult to collect

through interpreters…

• Professional simultaneous interpreters– Translate everything; maybe hire 2 interpreters?– Do not save money by hiring “cheap” interpreters

• Language Expansion– Interpreters have to keep up…

So you need to test in…

Make sure moderator speaks at an appropriate rate for interpreters

Interpreters & Translations• Observation Room:

• Interpreters should see participant’s face• What’s the ambient language in obs room?• Observers listening live in different languages?

• Translating test documents

– Consider double translation to check accuracy

(i.e., translate it back to native tongue)

Planning Fieldwork• Who should moderate?

– Native speakers only? Fluent non-native OK?

• What time of day can sessions be scheduled?

• How many sessions can the moderator do per day?

• National, cultural, religious issues…bugger

Finding vendors can be hard and can be easy, some strategies…– Ask colleagues and clients– Resource networks

• UPA • HCIRN • STC

– Search engines (google, yahoo, ask)

Vendor quality (over long-distance) is hard, but necessary; some ideas…– Check References– Ask for CVs of consultants– Ask about experience with

foreign clients– Recent clients

Connection of Cultures

Business practices– Confidentiality and

propriety not viewed similarly

– Project timelines and urgencies may be more “elastic”

– Budget, currency and payment terms can be challenging – always hidden costs (e.g., visas)

Connection of Cultures

So you need to test in…

Plan for a lot of time between sessions

Observers are Here

Testing is Here

Remote Observation Project team wants to watch

testing, but can’t travel? Stream sessions…requires

knowledge of network infrastructure and special tools– Encoded video pushed to

streaming server and viewed anywhere

Technical Requirements• Ship or Carry Equipment?

– Customs = Gotcha!

So you need to test in…

Careful what you pack for the lab

Technical Requirements• Ship or Carry Equipment?

– Customs = Gotcha!

• Even keyboards! AZERTY v QWERTY

• Powering equipment 220v or 110v or Both?– E.g., Sao Paulo has either 220 v or 110 v

plugs

• Local power…hopefully

So you need to test in…

Have a back up plan in case of power loss…or Internet…or monsoons

Facilities• Often unfamiliar and out of our control

– Facility quality is hard to judge

So you need to test in…

Some respondents won’t go to test facilities; test in hotels

Budgeting• Budgets are always up in the air with global

user research– Typically more expensive

• Then there are the extraordinary expenses…

So you need to test in…

Budget for security guards

Test Planning• A Test Plan identifies all of the points of the

testing:– Objectives– Research design– Participant screening criteria– blah… blah… blah…

• And Intercultural UX Research elements

Models and Methods for Global User ResearchRobert Schumacher, PhD.Managing DirectorUser Centric, Inc

bob@usercentric.com+1.630.320.3900

For a Better User Experience in call centers, Press 1http://www.usercentric.com/webinars/for-a-better-user-experience-in-call-centers-press-1

There are many ways to succeed There are many more ways to fail