Transcript
Page 1: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

HOW HAYMARKET IS ESTABLISHING !

A TESTING CULTUREJon Warden

Head of User Experience !

Shruti Ahuja Head of Analytics and Optimisation

Page 2: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

Jon Warden Head of User Experience!Haymarket Media Group !

uk.linkedin.com/in/jonwarden/!@jonwarden

My background!10 years at News UK in various design & UX roles 5 years as a UX consultant, freelancing at various organisations Haymarket to date: Initial set up, and 2 years management of internal UX team

Page 3: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

Shruti Ahuja Head of Analytics and Optimisation!Haymarket Media Group !

uk.linkedin.com/in/shrutiahuja/

My background!8 years experience in Analytics and Optimisation across Media, eCommerce, CE, Automotive, Travel and Financial services Haymarket to date: Develop analytical infrastructure across 20 B2C brands Haymarket this year: Expand Analytics to B2B brands and strengthen Optimisation structure for B2C brands

Page 4: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

– Grace Murray Hopper, Computer scientist and US Navy Admiral

“One accurate measurement is worth a thousand expert opinions”

Page 5: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

Who are Haymarket Media Group?

Page 6: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

AN INTERNATIONAL PUBLISHING HOUSE

• 72 market leading brands • 20 offices • Across six countries • 1500 + staff • Business Media division • Consumer Media division • For Print, Digital, Live Media and events

We create award winning specialist content for international audiences

Page 7: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

AN INTERNATIONAL PUBLISHING HOUSESome of our digital brands

Page 8: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

AN INTERNATIONAL PUBLISHING HOUSESome of our digital brands

Page 9: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

AN INTERNATIONAL PUBLISHING HOUSESome of our digital brands

Page 10: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

Our journey with Optimizely

Page 11: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

WITH OPTIMIZELY SINCE MAY 2013• We started off on a month-by-month Gold

package !

• We completed very basic experiments on: Whatcar.com PistonHeads.com CaravanSiteFinder.com

!

• We quickly found we were regularly hitting our traffic allocation

• In April 2014 we upgraded to Platinum

Page 12: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

WHY WE UPGRADED• We needed dedicated support • We needed unlimited projects, user accounts,

and more features to use on all our brands • Other areas of the business were interested in

using Optimizely, including international divisions • We now have dedicated accounts for:

Haymarket Business Media!Haymarket Consumer Media !PistonHeads.com

Page 13: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

The Optimisation process

Page 14: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

OPTIMISATION & BUSINESS PRACTICEEach company has their own process for A/B testing

Success/fail

Assumption/hypothesis reported to

Product team

Publisher/feature request

Business criteria UX / UI / FEcreate experiment

Analytics reports

Test runs for 2 weeks minimum

Product / UX / Analytics

define test

No

Yes

Test report created

Sent to business for approval

Budget agreed to create dev work for sprintPushed to live

Test fail

Test report created

Page 15: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

OPTIMISATION & BUSINESS PRACTICEEach test goes on the testing road map

Page 16: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

OPTIMISATION & BUSINESS PRACTICEEach test has a reporting document and test ID

Page 17: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

OPTIMISATION & BUSINESS PRACTICE

• The reports

Page 18: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

OPTIMISATION & BUSINESS PRACTICE

• The reports

Page 19: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

OPTIMISATION & BUSINESS PRACTICE

• The reports

Page 20: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

IN AN IDEAL WORLD…• Over all our process is a good one, but… • We’ve found that we’ve had a few issues • Maintaining focus, and learning is difficult, we’re a busy group • We don’t have a dedicated Optimisation team, yet! • Everyone has their own day jobs, business, analytics, product, ux, editorial • Resourcing - we have small analytics and UX teams • Business see the validity, but it isn’t business critical in certain disciplines • Getting budget released or time for iteration is difficult due to dev roadmap

Page 21: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

SOLUTIONS• We’re trying to change the mindset and the culture • We need to create an Optimisation process ‘all’ buy into • We’ve enlisted the help of Optimizely experts to help us • Getting onsite training for the team • Discussing testing strategies with senior management • In the past we’ve hired a third party CRO agency to get more focus • We need to create an in house dedicated team to focus efforts

Page 22: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

Experiments - what we’ve learnt

Page 23: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

WHAT WE’VE LEARNT

1. Do - ‘Start simple’, until you know what you’re doing 2. Do - Create as many tests as possible, and run them consecutively 3. Do - Measure the numbers, and then learn about the results 4. Do - Leave experiments to run as long as needed 5. Do - Embed a testing culture into your business 6. Do - Ask for help to learn more

Page 24: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

WHAT WE’VE LEARNT1. Don’t - run before you can walk, but learn as you go 2. Don’t - try experiments with too many goals 3. Don’t - create A/B and MVT tests with too many variants at first,

without having the proper knowledge 4. Don’t - launch multiple experiments on the same site, at the same

time, they have a tendency to clash with each other, causing experiments to fail

5. Don’t - forget to inform stakeholders you’re running tests

Page 25: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

WE’VE SIMPLIFIED OUR TESTS DOWN, !AND HAD SUCCESS

• We recently created an experiment on one of our sites • Using two styles and four variants

Variant A Variant B

Variant C Variant D

Page 26: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

WE’VE SIMPLIFIED OUR TESTS DOWN, !AND HAD SUCCESS

• The results were astounding! • Variant D had an uplift of +197.6%

Page 27: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

PISTONHEAD: ANALYSIS-LED OPTIMISATION

Shruti Ahuja Head of Analytics and Optimisation

Page 28: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

ABOUT PISTONHEADS.COM

• Connects car enthusiasts to dealers and one another

• An automotive classifieds platform

• One-stop car shop

Page 29: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

HOW DID WE CREATE OUR OPTIMISATION ROADMAP?

Page 30: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

WHAT IS OUR OPTIMISATION PROCESS?

Insights Generation

Roadmap Creation

Test ImplementationResults

Analysis

Learning

Page 31: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

WHAT DETERMINES OUR TESTING ROADMAP?

• Identify KPI’s that support the Business Model

• Investigate issues that hinder KPI conversion

• Generate ideas to test and improve performance on an on-going basis

!!!!!! Visitors

!!!!!! Inquiries

!!!!!! Leads

!!!!!! Sales

Page 32: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

WHAT INSPIRES OUR TEST IDEAS?

• Monthly dashboard reviews • Ad-hoc analysis • Product related issues

Page 33: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

OUR MOST SUCCESSFUL TESTObjective:!Determine if introducing a new layout with an inbuilt Contact Seller form and a large image increases site conversion !Hypothesis:!Currently users have a 2-step process to contacting sellers. The simplification of this and introduction of larger images will improve interaction with the ad details pages !Test Description:!A/B test. Control (Existing content: 2 step form & small images), Alternate B (In-page form, in-page images) !Success Metrics:!• Traffic • Retention • Contact Seller Conversion • Revenue

Page 34: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

OUR MOST SUCCESSFUL TEST

Page 35: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

WHEN TESTS FAIL…Objective:!Understand how user behaviour is affected by using a car like VRM on the parts and plates landing page. !Hypothesis:!The current parts and plates landing page design mirrors the old style SSI landing page. The new VRM form style has been seen to perform better for cars. This test will confirm that the design works for parts and plates too. !Test Description:!A/B test. Control, variation B - car-like VRM !Success Metrics:!Proceed to next step Ad creation conversion rate

Page 36: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

WHEN TESTS FAIL…

Page 37: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

WHAT WE LEARNT

• Optimisation is a learning process

• Never assume / preempt • You win some, you lose some • In either instance, test further!!

Page 38: Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

THANK YOU!QUESTIONS?


Top Related