growing your business through experimentation
TRANSCRIPT
Product, Design, Engineering, Marketing, Growth, Sales
What do you do?
How do you make all the things
better?!
Figuring this out is your job.
A/b testing
Only 1 out of 5 tests win.
4 out of 5 tests cost you money.
So, why experiment if so many tests fail?
Go Where Customers Are!Long, slow, SaaS ramp of death from 2006…
Go Where Customers Are!Now…
Go Where Customers Are!Growth can be sped up!
Increase Hello Bar’s installation rate
Improve the KISSmetrics features page
Increase signups from Crazy Egg’s homepage
Grow Crazy Egg’s revenue
Results from 3 months of focused growth effort
Results: KISSmetrics signups over time
Simple 4-step experimentation process
Data Figure out what to improve Hypothesis Make an educated guess Experiment Test your guess Act Decide what’s next
Data Figure out what to improve
Data doesn’t make decisions for you.
Data informs your decision making.
Data helps you identify the most painful problems people have. (not the ones that bother you the most)
Find problems worth solving by starting with quantitative data.
When in doubt, look at your top pages. Usually the homepage.
A simple framework for choosing what to improve
Find pages with high traffic and high bounce rate
HTTP://GOOGLE.COM/ANALYTICS
Increase Conversions By A/B Testing
Do a Happy Dance But Watch it Closely
Get More Traffic So You Can Start A/B Testing
Get More Traffic So You Can Do a Happy Dance
High Traffic
Low Traffic
High ConversionsLow Conversions
How To Figure Out What To Improve
Where do you have the most visitors, users, customers, emails, or notifications?
More volume helps you test faster.
๏ Website Traffic / App Installs
๏ Welcome Emails
๏ Web / Mobile Onboarding
๏ Email Digests
๏ Triggered Notifications
๏ Cart / Funnel Abandonment
๏ Dormant / Churned users
Common areas with large volume and high impact potential
What percentage of your customers or users have adopted each feature? Intercom on Product Management
When you’ve already got usage, analyze it.
Find out why by getting qualitative data.
A few qualitative tools
HTTP://USERTESTING.COM HTTP://SURVEYMONKEY.COM
A few more qualitative tools
HTTP://QUALAROO.COM HTTP://OLARK.COM
What frustrates people?
Find people who don’t do things and email them
EMAIL + SURVEYMONKEY
User Research gives you a headstart
Hypothesis Make an educated guess
Ideas Designs Solutions
Assumptions(A guess about the best way to solve a problem)}
Everything can start with a testable hypothesis.
Before you create anything, make an educated guess.
Use simple equations to assess potential impact of each idea
DAN MCKINLEY HTTP://KISS.LY/DANETSY
visits x conversion rate x order value x li4 = ?
users x order value x resurrec7on rate = ?
ESTIMATED
ESTIMATED
MODELED
MODELED
Assess the potential impact, difficulty and priority of each test idea you have.
Prioritize your tests
Experiment Test your guess
Test your hypothesis by starting an experiment.
Not button colors…
Don’t copy “best practices” blindly
Test specific button copy
Sensitivity tests help you learn what your visitors care about and what they don’t.
Run sensitivity tests on your web pages
Remove graphical elements and see what happens
Does social proof and logos matter for your visitors?
Remove social proof and learn what happens
Act Decide what’s next
You’ve probably seen an A/B testing report before
1. Your starting hypothesis.
2. Dates for when you ran the test.
3. Screenshots of control and all variations used during the test.
4. Expected change in conversion.
5. Probability of change.
6. Screenshot of the raw data used in your analysis.
7. A screenshot to the A/B Test Report in KISSmetrics.
8. The decision you've made from the test.
9. What you learned from the test.
Document every A/B test and create a playbook
What learnings do we want to carry over from the past?
Helps you avoid unknowingly decreasing your conversions.
✓ Keep the footer links
✓ Autofocus on the first form field
✓ Do not include social proof
✓ Link to the features page
✓ Use “Log in” copy on the button
Create experiment checklists based on learnings
If you’re not testing, you’re not learning.
☀
Always have tests running, no matter how small.
⌾
If you don't have a process, you won't continuously improve.
∞
Work on things that matter. Make those things better. Be happier ☺
