tactics for truly effective experiment design by grace ng - the lean startup conference 12/11/14

Designing EffectiveExperiments

Grace Ng | @uxceo

Test your business idea in just 5 minutes

Photo Q&A

What is this?

Minimum Viable Product An experiment that uses the least amount of effort

to get the most amount of customer feedback

Minimum Viable Product An experiment that uses the least amount of effort to test the

Riskiest Assumption against real customer feedback

Do people even have visual questions to post?

Riskiest Assumption: Have photo Qs to post.

Ask Question

Upload Photo

What kind of bird is this? (Is it even a bird??)

Will there be people who are able to answer the photo questions that are posted?

Riskiest Assumption: Someone can answer within given time.

What kind of bird is this? (Is it even a bird??)

Will this scale? Can this be replicated?

LEARN MEASURE

BUILD

I Don’t Care...

$2M. 2 Years.

Mistakes

Effective Interviews

1. Tested the wrong assumptions

Mistakes

Effective Interviews

2. Asked the wrong questions

Mistakes

Effective Interviews

3. Didn’t define success

Mistakes

Effective Interviews

4. Don’t need to build to test

Mistakes

Case Study

Founders UX Help

Experiment: 1

Customer: Bootstrapped startup founders

Experiment: 1

Customer: Bootstrapped startup founders

Problem: Getting UX feedback

Experiment: 1

Customer: Bootstrapped startup founders

Problem: Getting UX feedback

Solution:

Experiment: 1

Customer: Bootstrapped startup founders

Problem: Getting UX feedback

Solution: Bootstrapped startup founders

Riskiest Assumption: No reliable source

Experiment: 1

Customer: Bootstrapped startup founders

Problem: Getting UX feedback

Solution: Bootstrapped startup founders

Riskiest Assumption: No reliable source

Method & Success Criterion Interview. 6/10 frustrated

1. Cost• Project Cost• Opportunity Cost

2. Existing metrics• Business KPI• Specific user behavior

How to MEASURE success?How to measure success?

Experiment: 1

Customer: Bootstrapped startup founders

Problem: Getting UX feedback

Solution: Bootstrapped startup founders

Riskiest Assumption: No reliable source

Method & Success Criterion Interview. 6/10 frustrated

Exploration Interviews

Effective Interviews

1. I’m doing some research around...

2. Have you ever [experienced problem]?

3. Tell me about that time, what happened?

4. What did you do to resolve it?

5. How satisfied were you with the outcome?

6. If unsatisfied, how else have you tried to solve it?

Effective Interviews

Collect Data

• Level of pain/frustration

• Frequency of encounter with pain

• Existing behavior to solve problem

• Time/money spent searching for a solution

Collect Data

Experiment: 1

Result & Decision 9/12. Persevere

Learning Friends. Classes. Pay for UX help

Riskiest Assumption: No reliable source

Method & Success Criterion Interview. 6/10 frustrated

GET OUT OF THE BUILDING!

Experiment: 2

Customer: Bootstrapped startup founders

Problem: Getting UX feedback

Experiment: 2

Customer: Bootstrapped startup founders

Problem: Getting UX feedback

Solution: Virtual UX feedback

Riskiest Assumption: Will they pay?

Method & Success Criterion Pitch. 8/10 will pay

$10

@uxceo grace@QuickMVP.com

No Backend@uxceo

No Backend

Experiment: 2

Result & Decision 10/10 paid. Test 2nd side of market

Learning Don’t mind virtual. Price is cheap.

Riskiest Assumption: Will they pay?

Method & Success Criterion Pitch. 8/10 will pay

GET OUT OF THE BUILDING!

Experiment: 3

Customer: UX designers

Problem: Getting clients for consulting

Experiment: 3

Customer:

Problem:

Solution: Virtual UX feedback

Riskiest Assumption: Will consult virtually

Method & Success Criterion Pitch. 8/10 will sign up

UX designers

Getting clients for consulting

Seriously?

Experiment: 3

Result & Decision 0/10. Pivot!

Learning Focus on < 5 years. 5-10 years have established client base.

Riskiest Assumption: Will consult virtually

Method & Success Criterion Pitch. 8/10 will sign up

GET OUT OF THE BUILDING!

Experiment: 4

Customer: Transition to UX designers

Problem: Getting clients for consulting

Experiment: 4

Customer:

Problem:

Solution: Virtual UX feedback

Riskiest Assumption: Will consult virtually

Method & Success Criterion Pitch. 60% will sign up

Transition to UX designers

Getting clients for consulting

@uxceo grace@QuickMVP.com@uxceo

Pitch

Google Ads

No Backend

@uxceo grace@QuickMVP.com

The 5 Minute Technique

Quick MVP@uxceo grace@QuickMVP.com@uxceo

Quick MVP

3 Data Points@uxceo @nycuxpa #ngnycuxpa@uxceo

3 Data Points

Experiment: 4

Result & Decision 37/184 emails. Persevere *

Learning * Did not meet success metric, but visitors referred friends

Riskiest Assumption: Will consult virtually

Method & Success Criterion Pitch. 60% will sign up

GET OUT OF THE BUILDING!

Concierge@uxceo grace@QuickMVP.com

Customer paid $10

UX designer signed up

Next Step?

Concierge@uxceo grace@QuickMVP.com

Concierge

Customer paid $10

UX designer signed up

Experiment: 5

Customer: Bootstrapped startup founders

Problem: Getting UX feedback

Solution: Virtual UX feedback

Experiment: 5

Customer: Bootstrapped startup founders

Problem: Getting UX feedback

Solution: Virtual UX feedback

Riskiest Assumption: Online service is sufficient

Method & Success Criterion Concierge. 60% satisfied

Concierge@uxceo grace@QuickMVP.com

Concierge

Customer submits request

UX designer gets email and submits feedback

Manually@uxceo grace@QuickMVP.com

Manually

Customer submits request

UX designer gets email and submits feedback

Experiment: 5

Result & Decision 20% satisfied. Iterate

Learning Solved problem, but wanted more interaction with designer

Riskiest Assumption: Online service is sufficient

Method & Success Criterion Concierge. 60% satisfied

GET OUT OF THE BUILDING!

High PainLow Pain

Low Frequency

High Frequency

Focus

http://bit.ly/xB0D

Grace Ng | @uxceo

Test your business idea in just 5 minutes

Thank You!