Josh LaMar, Senior UX Researcher !

Email: joshlamar@me.comTwitter: @joshlamar

http://about.me/joshlamar http://linkedin.com/in/joshlamar

The Science and Art of Methodology

@joshlamar @pssigchi #newmethodologies

Agenda

• About Me

• The Science of Method

• The Art of Method

• Creating New Methods

• Breakout

• Closing and Q&A

My background in a nutshell

• B.A. in Music Composition and English Poetry

• M.S. in Human-Centered Design and Engineering

• People that inspire me:

• Galileo Galilei

• Leonardo DaVinci

• Andy Warhol

• Keith Haring

• Jack Kerouac

• Allen Ginsberg

• William Butler Yeats

• Claude Debussy

• Igor Stravinsky

The Science of Method

Photo: Galileo’s Telescopes

Elements of a method

• The Scientific Method

• Research Questions/Areas of inquiry

• Variables

• Participants

• Measurement

• Types of Research and Methodology Building Blocks

• Structuring a Study

The Scientific Method

• Who invented the telescope and the scientific method?

• Inspiration for all my studies: The Scientific Method (Simplified)

1. Research Question: What you want to investigate

2. Hypothesis: What you think is true

3. Experimentation: How you gather data

• True experiment testing with control and independent variable OR gathering user data through observation and discussion?

4. Analysis: How you discover what is actually true

• Each item is its own separate talk, but today: methods

Research Questions and Variables

• Research Question: Will the change in Version B cause users to be more successful when doing X?

• Hypothesis: Version B is better than Version A (better = higher success rate, faster time on task, higher satisfaction, etc.)

• Independent variables (cause or the thing being manipulated)

• E.g. Versions A (control) and B

• Dependent variables (effect)

• E.g. Higher success rate on Version B over Version A

Other Variables

• Location

• In a lab, on the street, in a house, at work, etc.

• You come to the participant: ethnography

• Participant comes to you: lab study

• Cost

• Recruiting costs, lab fees, gratuity fees, etc.

• Type of data

• Quantitative (Numbers)

• Qualitative (Everything else)

• Time

• How long do you have to run the study? A day, week, month, couple of months?

• How long is each session? 30 min,60 min, 90 min, 2 hours, more?

Participants

• Who are your target users?

• What do they currently use?

• How old are they?

• How tech savvy are they?

• Create a screener to get these people

• Number of Participants

• How many participants in each session?

• How many participants do you need?

• http://www.nngroup.com/articles/how-many-test-users/

How Many Users?

Number of Users How confident can you be?

3 Get a sense of big problems and broad themes, but not all of the issues. Be ok with missing some.

6 Feel good about finding the most important issues, you might miss some small things, but you’ll test again, right?

10Be pretty confident that you found all the major issues, some minor issues and even some differences in behaviors across

different groups.

12-16 For situations where it’s extremely important to be certain. You can also test iteratively over time.

Measurement

• Objective Performance Measures

• Observations/What users do

• Success rate

• Time on task

• Errors

• Subjective Perception Measures

• Pre and post surveys

• Things on Likert scales

• What users say/Quotes

• SUS (System Usability Scale)

Types of Research

Qualitative Research (Insights)

Quantitative Research (Validation)

Behaviors (What people do)

Goals/Attitudes (What people say) Eye Tracking

Usability Studies

Ethno/Field Visits Longitudinal/Diary

Competitive BenchmarkPersonas

Heuristic Evaluation Literature Reviews Expert Panel Customer Panel

Surveys

Customer FeedbackA/B Testing

Instrumentation

Interviews

Methodology Building Blocks

How do you address a research question?

1. Interview: How get people to talk a lot

2. Focus Group: How to get people not to dominate a conversation

3. Ethnography: How to get people to talk to you on their own turf

4. Usability Study: How to watch people not see the button

5. Concept Study/Cognitive Walkthrough: How to get people to say what they expect will happen *BEFORE* they click on it

6. Survey: How to see what a lot of people have to say about it

Examples

Method Number of Participants

Length of Session

1:1 or Group?

Qual or Quant? Location

Usability Study 10 2 hours 1:1 Both* Lab

Quick Usability

Study

5 1 hour 1:1 Both* Lab

Focus Group 10 2 hours Group Qual Lab or Conf room

Interview 10 2 hours 1:1 Qual Lab

Survey 300-1,000 NA NA Quant Home

Ethno 10 2 hours 1:1* Qual Home

Weekly Study Schedule

Mon Tue Wed Thur Fri

1 hour sessions

Gather Research Questions

Determine Method Prepare Run study

6 pptsAnalyze +

Report

2 hour sessions

Gather Questions

Method + Prep 3 ppts 3 ppts Analyze +

Report

Focus Group

Gather Questions

Method + Prep

10 ppts2 hours

x2 or moreAnalyze Report

Monthly Study Schedule

Mon Tue Wed Thur Fri

Week 1Gather

Research Questions

Develop Screener

Start recruiting

Meet with stakeholders

Week 2 Design Study Method

Research Plan Review

Week 3 Pilot: 1 ppt2 hours

3 ppts2 hours

3 ppts2 hours

3ppts2 hours

2 ppts2 hours

Week 4 Analyze Data Analyze Data Analyze Data Analyze Data Report Findings

Structuring a Study/How-to Frankenstudy

• Pre-survey - What they think now, before you talk about what you want to know

• Behavioral interview - So you can get what they say out of the way before it’s influenced by what they do

• Usability Tasks - Version A - So you can watch what they do

• Usability Tasks - Version B (Counterbalance)

• Conceptual tasks/cognitive walk through - To see how users think it will work

• Feature prioritization - Having users rate which things are most important to them

• Post-survey - So you have data to compare with the pre-survey

Daily Schedule with 2 Researchers

Lab 1 Lab 28 am9 am P110 am P2 P111 am P2

12 noon P31 pm P4 P32 pm P43 pm P54 pm P6 P55 pm P66 pm Debrief Debrief

The Art of Method

Photo: Galileo’s drawings of the moon, The Starry Messenger, 1610

Back to Galileo…

Galileo’s drawing of the stars of Orion and the Pleiades

Sidereus Nuncius, The Starry Messenger

Galileo’s drawings of the moons of Jupiter

Sidereus Nuncius, The Starry Messenger

The Copernican System

Also known as the heliocentric system, the sun, sol, is at the center instead of the Earth (Ptolemaic/geocentric, system)

Dialogue Concerning the Two Chief World Systems: Aristotle, Ptolemy, and Copernicus

Lessons from Galileo

• Question what is known to be “True”

• Use your own eyes to observe and your brain to reason

• Use the Scientific Method to build on theories and discover Truth

• Not everyone will believe you at first, but over time, the evidence will surmount, a crisis will occur, and a new paradigm will emerge

• See also, Thomas Kuhn’s “The Structure of Scientific Revolutions”

• Despite other scientists, philosophers, and theologians all being against him, Galileo stuck with his gut and what he was “Visually certain” was true because he saw it with his own eyes

Constraints

“You can pick two of the three: it can be good, it can be cheap, or it can be fast. You can’t have all three.” - Corey, One of the best PMs I’ve worked with

• It can be good and fast, but it won’t be cheap

• It can be fast and cheap, but it won’t be good

• It can be good and cheap, but it won’t be fast

We can also look at this in terms of constraints of time, cost, and rigor.

The Problem with Constraints

The Problem: Leadership thinks all three are possible. We really want all three to be possible…

Reality: You can get two of the three, not all three, unless you’re magic.*

*If you are a wizard, come talk to me later.

What to do about it? Understand the constraint and work around it. In Science: be aware of your lens and reduce threats to internal validity of your study. Find other data points from other studies and other sources to support your findings.

Constraint 1: Time

• It’s hard to do something well quickly

• Eliminate distractions when you need to focus

• It takes time to recruit participants and plan studies

• Recruit ahead of time, knowing you will have something to test

• Take notes and track key metrics while you’re running the study so you have all the data points you need as soon as the study is over

• Debrief session right after your last participant

• Have a Quick Findings email template ready to go to send out the day after your study

• Methods that optimize for time: Quick Pulse/Mini Usability Study, Surveys, Heuristic Evaluations, Guerrilla Studies

Constraint 2: Cost

• You can’t get around it, but you can be creative about it

• Recruiting in a short period of time costs more, so if you plan ahead, it can reduce costs that way

• Online options like surveys and usertesting.com are also less expensive options

• If all else fails: a stack of Starbucks gift cards can entice strangers to talk to you

Constraint 3: Rigor

• How sure are you that your results are good?

• Do you have the right participants? or are they not representative users?

• Do you have enough* participants to see the patterns?

• Are you looking for statistical significance?

• Do multiple data sources point to the same conclusion?

• Do external sources (lit review) also align?

• Does the study need to be repeated to be certain of the findings?

• How certain do you need to be?

– Igor Stravinsky, Poetics of Music

“My freedom will be so much the greater and more meaningful the more narrowly I limit my field of action and the more I surround myself

with obstacles. Whatever diminishes constraint diminishes strength. The more constraints one imposes, the more one frees one's self of the chains that shackle the spirit.”

Stravinsky’s The Rite of Spring, 1913

1940

Do not go gentle into that good night, Dylan Thomas

Freedom in Constraints

• Stravinsky created The Rite of Spring out of constraints of a new scale (Octatonic)

• Dylan Thomas wrote “Do not go gentle into that good night” out of major constraints of rhyme and meter

• Galileo proved the heliocentric theory despite constraints… such as the Inquisition

Inspiration

• Austin Kleon talks about climbing up your philosophical family tree

• Who inspires you? What is it about them that inspires you?

• How can you take these people and fit them into your work? your life?

• I give each of you permission to be inspired by the things that you are inspired by. Read a book about them, listen to them, look at their work. Whatever it takes.

Creating New Methods

“If I have seen farther, it is by standing on the shoulders of giants.” - Isaac Newton, Letter to Robert Hooke, 1676

Creating New Methods

• Every new method is just a recombination of different pieces

• Once you know the pieces, you can recombine them in different ways to meet different goals

• You’re not starting from zero, build on what we already have… find those giants and stand on their shoulders

• Don’t try to create in a vacuum: involve others, bounce ideas off others, share your ideas

• See also: Austin Kleon, “Steal like an Artist”

• Take inspiration from anywhere that is meaningful to you: Science, Art, Literature, Pop Culture, etc.

Case Study #1: The Mini Study

• Business Need: Quick, agile research on an ad hoc basis; new things to test all the time

• Existing Method: Quick Pulse Study: 3 users, in 2 hour sessions scheduled every Thursday, 4 weeks a month

• Constraints: 3 users a week isn’t enough to feel confident in the findings, budget for only 12 users per month

• Response: The Mini Study: 6 users, 2 hour sessions run over two days instead of one day; twice a month instead of 4x a month

Case Study #2: The Pepsi Challenge

• Business Need: Understand how our apps stand up against the steepest competition

• Existing Method: Counterbalanced Competitive Benchmark Study: Usability + Survey

• Constraints: Time and adoption of user research into a new organization

• Inspiration: The Pepsi Challenge: Inspired by the Pepsi Taste Test from the 1980s, the idea was to try out an app, rate what you think about it, try out the other app, rate it, and then talk about what you liked better and why

• Impact: The Pepsi Challenge is used by leadership to evaluate our apps and has become a key research metric in the division

Case Study #3: STAGE

• Business Need: Test a lot of apps very quickly

• Existing Methods: Focus Group + Independent exploration + Survey + Focus Group

• Constraints: Time and enough users to feel confident

• Response: Solo Trial and Group Evaluation: 7 apps, 7 2-hour sessions, 6 users per session = 42 users

• Impact: Very quick methodology to evaluate apps with minimal overhead; scorecard of evaluation of app vs. current app

A formula of sorts

Business Need/Question

+ Existing Methods (Science)

+ Constraints (Art)

+ Inspiration

___________________________

= New Methodologies

Breakout Time: That’s right, you have to talk to each other.

Breakout

• Scientists and Researchers: Split up, you are now the captains

• Artists, Designers, and others: Find a group, preferably with someone you don’t know

• The Task:

• Each group takes a current question you want to address or business need

• Pick a constraint, or a few (Time, cost, or rigor/participants)

• Add inspiration

• Recombine existing methods

• We’ll reconvene in 10 min to share what you came up with

What did you learn?

Resources• Galileo, A Very Short Introduction, by Stillman Drake

• Steal Like An Artist, by Austin Kleon

• The Structure of Scientific Revolutions, by Thomas Kuhn

• Beautiful Evidence, by Edward Tufte

• Photo Credits:

• http://en.wikipedia.org/wiki/Galileo_Galilei

• http://en.wikipedia.org/wiki/Sidereus_Nuncius

• http://www.nasa.gov/

• Other sites referenced

• http://en.wikipedia.org/wiki/Villanelle

Thank you. Q&A

Email: joshlamar@me.comTwitter: @joshlamar http://about.me/joshlamar http://linkedin.com/in/joshlamar