Download - Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014
HOW HAYMARKET IS ESTABLISHING !
A TESTING CULTUREJon Warden
Head of User Experience !
Shruti Ahuja Head of Analytics and Optimisation
Jon Warden Head of User Experience!Haymarket Media Group !
uk.linkedin.com/in/jonwarden/!@jonwarden
My background!10 years at News UK in various design & UX roles 5 years as a UX consultant, freelancing at various organisations Haymarket to date: Initial set up, and 2 years management of internal UX team
Shruti Ahuja Head of Analytics and Optimisation!Haymarket Media Group !
uk.linkedin.com/in/shrutiahuja/
My background!8 years experience in Analytics and Optimisation across Media, eCommerce, CE, Automotive, Travel and Financial services Haymarket to date: Develop analytical infrastructure across 20 B2C brands Haymarket this year: Expand Analytics to B2B brands and strengthen Optimisation structure for B2C brands
– Grace Murray Hopper, Computer scientist and US Navy Admiral
“One accurate measurement is worth a thousand expert opinions”
Who are Haymarket Media Group?
AN INTERNATIONAL PUBLISHING HOUSE
• 72 market leading brands • 20 offices • Across six countries • 1500 + staff • Business Media division • Consumer Media division • For Print, Digital, Live Media and events
We create award winning specialist content for international audiences
AN INTERNATIONAL PUBLISHING HOUSESome of our digital brands
AN INTERNATIONAL PUBLISHING HOUSESome of our digital brands
AN INTERNATIONAL PUBLISHING HOUSESome of our digital brands
Our journey with Optimizely
WITH OPTIMIZELY SINCE MAY 2013• We started off on a month-by-month Gold
package !
• We completed very basic experiments on: Whatcar.com PistonHeads.com CaravanSiteFinder.com
!
• We quickly found we were regularly hitting our traffic allocation
• In April 2014 we upgraded to Platinum
WHY WE UPGRADED• We needed dedicated support • We needed unlimited projects, user accounts,
and more features to use on all our brands • Other areas of the business were interested in
using Optimizely, including international divisions • We now have dedicated accounts for:
Haymarket Business Media!Haymarket Consumer Media !PistonHeads.com
The Optimisation process
OPTIMISATION & BUSINESS PRACTICEEach company has their own process for A/B testing
Success/fail
Assumption/hypothesis reported to
Product team
Publisher/feature request
Business criteria UX / UI / FEcreate experiment
Analytics reports
Test runs for 2 weeks minimum
Product / UX / Analytics
define test
No
Yes
Test report created
Sent to business for approval
Budget agreed to create dev work for sprintPushed to live
Test fail
Test report created
OPTIMISATION & BUSINESS PRACTICEEach test goes on the testing road map
OPTIMISATION & BUSINESS PRACTICEEach test has a reporting document and test ID
OPTIMISATION & BUSINESS PRACTICE
• The reports
OPTIMISATION & BUSINESS PRACTICE
• The reports
OPTIMISATION & BUSINESS PRACTICE
• The reports
IN AN IDEAL WORLD…• Over all our process is a good one, but… • We’ve found that we’ve had a few issues • Maintaining focus, and learning is difficult, we’re a busy group • We don’t have a dedicated Optimisation team, yet! • Everyone has their own day jobs, business, analytics, product, ux, editorial • Resourcing - we have small analytics and UX teams • Business see the validity, but it isn’t business critical in certain disciplines • Getting budget released or time for iteration is difficult due to dev roadmap
SOLUTIONS• We’re trying to change the mindset and the culture • We need to create an Optimisation process ‘all’ buy into • We’ve enlisted the help of Optimizely experts to help us • Getting onsite training for the team • Discussing testing strategies with senior management • In the past we’ve hired a third party CRO agency to get more focus • We need to create an in house dedicated team to focus efforts
Experiments - what we’ve learnt
WHAT WE’VE LEARNT
1. Do - ‘Start simple’, until you know what you’re doing 2. Do - Create as many tests as possible, and run them consecutively 3. Do - Measure the numbers, and then learn about the results 4. Do - Leave experiments to run as long as needed 5. Do - Embed a testing culture into your business 6. Do - Ask for help to learn more
WHAT WE’VE LEARNT1. Don’t - run before you can walk, but learn as you go 2. Don’t - try experiments with too many goals 3. Don’t - create A/B and MVT tests with too many variants at first,
without having the proper knowledge 4. Don’t - launch multiple experiments on the same site, at the same
time, they have a tendency to clash with each other, causing experiments to fail
5. Don’t - forget to inform stakeholders you’re running tests
WE’VE SIMPLIFIED OUR TESTS DOWN, !AND HAD SUCCESS
• We recently created an experiment on one of our sites • Using two styles and four variants
Variant A Variant B
Variant C Variant D
WE’VE SIMPLIFIED OUR TESTS DOWN, !AND HAD SUCCESS
• The results were astounding! • Variant D had an uplift of +197.6%
PISTONHEAD: ANALYSIS-LED OPTIMISATION
Shruti Ahuja Head of Analytics and Optimisation
ABOUT PISTONHEADS.COM
• Connects car enthusiasts to dealers and one another
• An automotive classifieds platform
• One-stop car shop
HOW DID WE CREATE OUR OPTIMISATION ROADMAP?
WHAT IS OUR OPTIMISATION PROCESS?
Insights Generation
Roadmap Creation
Test ImplementationResults
Analysis
Learning
WHAT DETERMINES OUR TESTING ROADMAP?
• Identify KPI’s that support the Business Model
• Investigate issues that hinder KPI conversion
• Generate ideas to test and improve performance on an on-going basis
!!!!!! Visitors
!!!!!! Inquiries
!!!!!! Leads
!!!!!! Sales
WHAT INSPIRES OUR TEST IDEAS?
• Monthly dashboard reviews • Ad-hoc analysis • Product related issues
OUR MOST SUCCESSFUL TESTObjective:!Determine if introducing a new layout with an inbuilt Contact Seller form and a large image increases site conversion !Hypothesis:!Currently users have a 2-step process to contacting sellers. The simplification of this and introduction of larger images will improve interaction with the ad details pages !Test Description:!A/B test. Control (Existing content: 2 step form & small images), Alternate B (In-page form, in-page images) !Success Metrics:!• Traffic • Retention • Contact Seller Conversion • Revenue
OUR MOST SUCCESSFUL TEST
WHEN TESTS FAIL…Objective:!Understand how user behaviour is affected by using a car like VRM on the parts and plates landing page. !Hypothesis:!The current parts and plates landing page design mirrors the old style SSI landing page. The new VRM form style has been seen to perform better for cars. This test will confirm that the design works for parts and plates too. !Test Description:!A/B test. Control, variation B - car-like VRM !Success Metrics:!Proceed to next step Ad creation conversion rate
WHEN TESTS FAIL…
WHAT WE LEARNT
• Optimisation is a learning process
• Never assume / preempt • You win some, you lose some • In either instance, test further!!
THANK YOU!QUESTIONS?