case study by jon warden and shruti ahuja of haymarket - optimizely experience london 2014

Download Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014

Post on 02-Dec-2014

241 views

Category:

Marketing

2 download

Embed Size (px)

DESCRIPTION

Jon Warden and Shruti Ahuja share their story how they established a Testing Culture at Haymarket during the Optimizely Experience London 2014.

TRANSCRIPT

  • 1. HOW HAYMARKET IS ESTABLISHING ! A TESTING CULTURE Jon Warden Head of User Experience ! Shruti Ahuja Head of Analytics and Optimisation
  • 2. Jon Warden Head of User Experience! Haymarket Media Group ! uk.linkedin.com/in/jonwarden/! @jonwarden My background! 10 years at News UK in various design & UX roles 5 years as a UX consultant, freelancing at various organisations Haymarket to date: Initial set up, and 2 years management of internal UX team
  • 3. Shruti Ahuja Head of Analytics and Optimisation! Haymarket Media Group ! uk.linkedin.com/in/shrutiahuja/ My background! 8 years experience in Analytics and Optimisation across Media, eCommerce, CE, Automotive, Travel and Financial services Haymarket to date: Develop analytical infrastructure across 20 B2C brands Haymarket this year: Expand Analytics to B2B brands and strengthen Optimisation structure for B2C brands
  • 4. One accurate measurement is worth a thousand expert opinions Grace Murray Hopper, Computer scientist and US Navy Admiral
  • 5. Who are Haymarket Media Group?
  • 6. AN INTERNATIONAL PUBLISHING HOUSE We create award winning specialist content for international audiences 72 market leading brands 20 offices Across six countries 1500 + staff Business Media division Consumer Media division For Print, Digital, Live Media and events
  • 7. AN INTERNATIONAL PUBLISHING HOUSE Some of our digital brands
  • 8. AN INTERNATIONAL PUBLISHING HOUSE Some of our digital brands
  • 9. AN INTERNATIONAL PUBLISHING HOUSE Some of our digital brands
  • 10. Our journey with Optimizely
  • 11. WITH OPTIMIZELY SINCE MAY 2013 We started off on a month-by-month Gold package ! We completed very basic experiments on: Whatcar.com PistonHeads.com CaravanSiteFinder.com ! We quickly found we were regularly hitting our traffic allocation In April 2014 we upgraded to Platinum
  • 12. WHY WE UPGRADED We needed dedicated support We needed unlimited projects, user accounts, and more features to use on all our brands Other areas of the business were interested in using Optimizely, including international divisions We now have dedicated accounts for: Haymarket Business Media! Haymarket Consumer Media ! PistonHeads.com
  • 13. The Optimisation process
  • 14. OPTIMISATION & BUSINESS PRACTICE Each company has their own process for A/B testing Test report created Success/fail Assumption/ hypothesis reported to Product team Publisher/feature request Business criteria UX / UI / FE create experiment Analytics reports Test runs for 2 weeks minimum Product / UX / Analytics define test No Yes Test report created Sent to business for approval Budget agreed to create dev Pushed to live work for sprint Test fail
  • 15. OPTIMISATION & BUSINESS PRACTICE Each test goes on the testing road map
  • 16. OPTIMISATION & BUSINESS PRACTICE Each test has a reporting document and test ID
  • 17. OPTIMISATION & BUSINESS PRACTICE The reports
  • 18. OPTIMISATION & BUSINESS PRACTICE The reports
  • 19. OPTIMISATION & BUSINESS PRACTICE The reports
  • 20. IN AN IDEAL WORLD Over all our process is a good one, but Weve found that weve had a few issues Maintaining focus, and learning is difficult, were a busy group We dont have a dedicated Optimisation team, yet! Everyone has their own day jobs, business, analytics, product, ux, editorial Resourcing - we have small analytics and UX teams Business see the validity, but it isnt business critical in certain disciplines Getting budget released or time for iteration is difficult due to dev roadmap
  • 21. SOLUTIONS Were trying to change the mindset and the culture We need to create an Optimisation process all buy into Weve enlisted the help of Optimizely experts to help us Getting onsite training for the team Discussing testing strategies with senior management In the past weve hired a third party CRO agency to get more focus We need to create an in house dedicated team to focus efforts
  • 22. Experiments - what weve learnt
  • 23. WHAT WEVE LEARNT 1. Do - Start simple, until you know what youre doing 2. Do - Create as many tests as possible, and run them consecutively 3. Do - Measure the numbers, and then learn about the results 4. Do - Leave experiments to run as long as needed 5. Do - Embed a testing culture into your business 6. Do - Ask for help to learn more
  • 24. WHAT WEVE LEARNT 1. Dont - run before you can walk, but learn as you go 2. Dont - try experiments with too many goals 3. Dont - create A/B and MVT tests with too many variants at first, without having the proper knowledge 4. Dont - launch multiple experiments on the same site, at the same time, they have a tendency to clash with each other, causing experiments to fail 5. Dont - forget to inform stakeholders youre running tests
  • 25. WEVE SIMPLIFIED OUR TESTS DOWN, ! AND HAD SUCCESS We recently created an experiment on one of our sites Using two styles and four variants Variant A Variant B Variant C Variant D
  • 26. WEVE SIMPLIFIED OUR TESTS DOWN, ! AND HAD SUCCESS The results were astounding! Variant D had an uplift of +197.6%
  • 27. PISTONHEAD: ANALYSIS-LED OPTIMISATION Shruti Ahuja Head of Analytics and Optimisation
  • 28. ABOUT PISTONHEADS.COM Connects car enthusiasts to dealers and one another An automotive classifieds platform One-stop car shop
  • 29. HOW DID WE CREATE OUR OPTIMISATION ROADMAP?
  • 30. WHAT IS OUR OPTIMISATION PROCESS? Insights Generation Roadmap Creation Test Learning Results Implementation Analysis