tactics for truly effective experiment design by grace ng - the lean startup conference 12/11/14
TRANSCRIPT
Designing EffectiveExperiments
Grace Ng | @uxceo
Test your business idea in just 5 minutes
Photo Q&A
What is this?
Minimum Viable Product An experiment that uses the least amount of effort
to get the most amount of customer feedback
Minimum Viable Product An experiment that uses the least amount of effort to test the
Riskiest Assumption against real customer feedback
Do people even have visual questions to post?
Riskiest Assumption: Have photo Qs to post.
Ask Question
Upload Photo
What kind of bird is this? (Is it even a bird??)
Will there be people who are able to answer the photo questions that are posted?
Riskiest Assumption: Someone can answer within given time.
What kind of bird is this? (Is it even a bird??)
Will this scale? Can this be replicated?
LEARN MEASURE
BUILD
I Don’t Care...
$2M. 2 Years.
Mistakes
Effective Interviews
1. Tested the wrong assumptions
Mistakes
Effective Interviews
2. Asked the wrong questions
Mistakes
Effective Interviews
3. Didn’t define success
Mistakes
Effective Interviews
4. Don’t need to build to test
Mistakes
Case Study
Founders UX Help
Experiment: 1
Customer: Bootstrapped startup founders
Experiment: 1
Customer: Bootstrapped startup founders
Problem: Getting UX feedback
Experiment: 1
Customer: Bootstrapped startup founders
Problem: Getting UX feedback
Solution:
Experiment: 1
Customer: Bootstrapped startup founders
Problem: Getting UX feedback
Solution: Bootstrapped startup founders
Riskiest Assumption: No reliable source
Experiment: 1
Customer: Bootstrapped startup founders
Problem: Getting UX feedback
Solution: Bootstrapped startup founders
Riskiest Assumption: No reliable source
Method & Success Criterion Interview. 6/10 frustrated
1. Cost• Project Cost• Opportunity Cost
2. Existing metrics• Business KPI• Specific user behavior
How to MEASURE success?How to measure success?
Experiment: 1
Customer: Bootstrapped startup founders
Problem: Getting UX feedback
Solution: Bootstrapped startup founders
Riskiest Assumption: No reliable source
Method & Success Criterion Interview. 6/10 frustrated
Exploration Interviews
Effective Interviews
1. I’m doing some research around...
2. Have you ever [experienced problem]?
3. Tell me about that time, what happened?
4. What did you do to resolve it?
5. How satisfied were you with the outcome?
6. If unsatisfied, how else have you tried to solve it?
Effective Interviews
Collect Data
• Level of pain/frustration
• Frequency of encounter with pain
• Existing behavior to solve problem
• Time/money spent searching for a solution
Collect Data
Experiment: 1
Result & Decision 9/12. Persevere
Learning Friends. Classes. Pay for UX help
Riskiest Assumption: No reliable source
Method & Success Criterion Interview. 6/10 frustrated
GET OUT OF THE BUILDING!
Experiment: 2
Customer: Bootstrapped startup founders
Problem: Getting UX feedback
Experiment: 2
Customer: Bootstrapped startup founders
Problem: Getting UX feedback
Solution: Virtual UX feedback
Riskiest Assumption: Will they pay?
Method & Success Criterion Pitch. 8/10 will pay
$10
Experiment: 2
Result & Decision 10/10 paid. Test 2nd side of market
Learning Don’t mind virtual. Price is cheap.
Riskiest Assumption: Will they pay?
Method & Success Criterion Pitch. 8/10 will pay
GET OUT OF THE BUILDING!
Experiment: 3
Customer: UX designers
Problem: Getting clients for consulting
Experiment: 3
Customer:
Problem:
Solution: Virtual UX feedback
Riskiest Assumption: Will consult virtually
Method & Success Criterion Pitch. 8/10 will sign up
UX designers
Getting clients for consulting
Seriously?
Experiment: 3
Result & Decision 0/10. Pivot!
Learning Focus on < 5 years. 5-10 years have established client base.
Riskiest Assumption: Will consult virtually
Method & Success Criterion Pitch. 8/10 will sign up
GET OUT OF THE BUILDING!
Experiment: 4
Customer: Transition to UX designers
Problem: Getting clients for consulting
Experiment: 4
Customer:
Problem:
Solution: Virtual UX feedback
Riskiest Assumption: Will consult virtually
Method & Success Criterion Pitch. 60% will sign up
Transition to UX designers
Getting clients for consulting
Google Ads
No Backend
3 Data Points@uxceo @nycuxpa #ngnycuxpa@uxceo
3 Data Points
Experiment: 4
Result & Decision 37/184 emails. Persevere *
Learning * Did not meet success metric, but visitors referred friends
Riskiest Assumption: Will consult virtually
Method & Success Criterion Pitch. 60% will sign up
GET OUT OF THE BUILDING!
Experiment: 5
Customer: Bootstrapped startup founders
Problem: Getting UX feedback
Solution: Virtual UX feedback
Experiment: 5
Customer: Bootstrapped startup founders
Problem: Getting UX feedback
Solution: Virtual UX feedback
Riskiest Assumption: Online service is sufficient
Method & Success Criterion Concierge. 60% satisfied
Concierge@uxceo [email protected]
Concierge
Customer submits request
UX designer gets email and submits feedback
Manually@uxceo [email protected]
Manually
Customer submits request
UX designer gets email and submits feedback
Experiment: 5
Result & Decision 20% satisfied. Iterate
Learning Solved problem, but wanted more interaction with designer
Riskiest Assumption: Online service is sufficient
Method & Success Criterion Concierge. 60% satisfied
GET OUT OF THE BUILDING!
High PainLow Pain
Low Frequency
High Frequency
Focus
Grace Ng | @uxceo
Test your business idea in just 5 minutes
Thank You!