emerce conversion
TRANSCRIPT
What we learned from running 200 tests a year?
Martijn Scheijbeler / Director of Marketing / [email protected] / @MartijnSch
Martijn Scheijbeler
Marketing Director at The Next WebFormer Lead SEO & Analytics
[email protected] @MartijnSch
INDEX
20,000 140,000 6,000visitors startups listed square meters
7,500,000unique visitors
INDEX.COConference
Testing Figures Data from 2015
2,5 CRO Team in 2015
38.000.000 Sessions with Tests
0 CRO Team in 2014
5-7 Tests per week
Priceless Amounts of fun
35% Traffic v.s. Tests
@MartijnSch
“If you double the number of experiments you’re going to double your inventiveness”
- Jeff Bezos
@MartijnSch
The opportunity is 350-400 tests a
year. As ROI per test is low we
need to run a lot to have a positive
return.
Efficiency <> Costs
@MartijnSch
SO WE STARTED TESTING WITH GOOGLE ANALYTICS CONTENT EXPERIMENTS
GOOGLE TAG MANAGER IS MAKING OUR LIVES EASIER + SCALABLE
FIND THE PERFECT PROCESS TO FIX EVERYTHING > ALWAYS
WHY? MORE EFFICIENCY: THE MORE TESTS, THE HIGHER THE INVENTIVENESS
We didn’t move to any ‘real’
CRO tool but decided to
iterate on our own: GTM <> JS.
GTM
@MartijnSch
Download:
Github.com/MartijnSch/CRO
Building our own solutions for
saving testing documentation
+ analysis of tests.
Innovate
@MartijnSch
We moved most of our
communication towards
Slack. Making the teams more
involved in the whole process.
Always communicate
@MartijnSch
We hired two front-end
developers, to make sure in
the long-run we can test
bigger things.
Front-end Dev
@MartijnSch
Improvements
5-7 Tests per week
2-4 Tests per week
30-40 Tests per year
20% More efficient
@MartijnSch
TLDR: Process
1. Have the right company vision 2. Have the right team structure 3. Have access to top level 4. Create partnerships with like minded 5. You’ll have trouble as well, nobody’s perfect 6. Don’t be afraid to invent the wheel again 7. Always focus on innovation
@MartijnSch
• Variant increase clicks with 16% • Had a significance + power level of > 99% • We ran the test for 1x 7+ days • Targetting over 200.000 users
Winner: Variant
Original:
Variant
Engagement is important,
focus on getting more clicks
on the related stories in the
sidebar.
Rankings!
• Variant increase clicks with 36% • Had a significance + power level of > 99% • We ran the test for 7+ days • Targetting over 300.000 users
Winner: Variant
Original: Variant:
Engagement is important,
focus on getting more clicks
on the related stories below
the article.
Related Stories
• Variant increase clicks with 36% • Had a significance + power level of > 99% • We ran the test for 2x 14+ days • Targetting over 500.000 users
Winner: Variant
Original:Variant
So what if we remove the
whole sidebar, wouldn’t people
be more focused on the stuff
that is out there?
Big, bigger
• Variant decreased clicks with 17% + 14% • Had a significance + power level of > 99% • We ran the test for 14+ days • Targetting over 420.000 users
Winner: Original
TLDR: Test learnings
1. Know what you’re testing 2. Focus on your user segments 3. Iterate on what you’re already testing 4. Focus on high significance 5. Test big, bigger, biggest!
@MartijnSch
“Well, if this guy wouldn’t have stuck
his fingers into a lawn mower, YES!”
Missing out on ±20 tests in 2015.
The Lawn Mower
Take Aways:
1. Focus on running X tests a year 2. Improve the process
3. Have the right (free) tools
4. Have the right people
5. Document/Communicate everything 6. Know what the future looks like 7. Have fun!
@MartijnSch