A/B Testing - How to Break Things and Fail Fast... Without Breaking Things

Download A/B Testing - How to Break Things and Fail Fast... Without Breaking Things

Post on 05-Dec-2014

152 views

Category:

Marketing

0 download

DESCRIPTION

Everyone wants to optimize their web property, especially if that web property is their only sales channel. Lots of tools like Optimizely, allow marketers to go in and run tests on their own in an effort to "break things and fail fast", then take those learnings to make the site better. Unfortunately, the risk of severely breaking things can be significant, and it isn't the marketer who gets the call in the middle of the night when something is broken. In this talk Mark will explore how Atlassian, the creator of JIRA and Confluence, has used Magnolia CMS and other tools to operationalize A/B testing to continually optimize Atlassians website. Using this process we can balance failing fast with keeping things running smoothly.

TRANSCRIPT

  • 1. MARK HALVORSON HEAD OF INTERACTIVE ATLASSIAN SOFTWARE @halv0112 A/B Testing HOW TO BREAK THINGS AND FAIL FAST WITHOUT BREAKING THINGS
  • 2. About Atlassian Products include: JIRA, Conuence, Bitbucket, HipChat Founded in 2002 Over 35,000 Customers Over 700 employees globally No sales people ! ! ! We h e l p g r e a t t e a m s b u i l d b e t t e r s o f t w a r e , t o g e t h e r.
  • 3. Think big, act small, fail fast, learn rapidly. "LEAN SOFTWARE DEVELOPMENT BY MARY AND TOM POPPENDIECK (2003)
  • 4. Atlassian A/B Testing is cross functional Email Product Management Customer Platform Customer Advocates Internal Systems ! ! ! C e n t r a l G r o w t h H a c k i n g Te a m t o s u p p o r t : Leads Product Marketing Support Bitbucket HipChat
  • 5. SCHEDULE TO AVOID CONFLICTS BUILD OUT GOOD CANDIDATES TRIAGE BASED ON POTENTIAL IMPACT COLLECT GREAT IDEAS Our Process ANALYZE RESULTS RUN EXPERIMENTS
  • 6. SCHEDULE TO AVOID CONFLICTS BUILD OUT GOOD CANDIDATES TRIAGE BASED ON POTENTIAL IMPACT COLLECT GREAT IDEAS Our Process ANALYZE RESULTS RUN EXPERIMENTS
  • 7. Where do experiment ideas come from?
  • 8. Anywhere.
  • 9. What are you are trying to improve?
  • 10. Blimp Dashboard
  • 11. JIRA Ticket
  • 12. Pebbles Change in copy or CTA presentation Rocks New page (e.g. no Dev Tools family page) Boulders Simplied order form. ! ! ! Types of tests
  • 13. Examples
  • 14. Reduce Friction
  • 15. Increase Visibility
  • 16. Test drive campaigns
  • 17. Simplify
  • 18. SCHEDULE TO AVOID CONFLICTS BUILD OUT GOOD CANDIDATES TRIAGE BASED ON POTENTIAL IMPACT COLLECT GREAT IDEAS Our Process ANALYZE RESULTS RUN EXPERIMENTS
  • 19. What are you are trying to improve?
  • 20. Blimp Dashboard
  • 21. JIRA Prioritized Backlog
  • 22. JIRA Ticket
  • 23. JIRA Ticket
  • 24. Experiment Illuminati
  • 25. SCHEDULE TO AVOID CONFLICTS BUILD OUT GOOD CANDIDATES TRIAGE BASED ON POTENTIAL IMPACT COLLECT GREAT IDEAS Our Process ANALYZE RESULTS RUN EXPERIMENTS
  • 26. Optimizely Build
  • 27. Optimizely Build
  • 28. Prepare for results.
  • 29. Confluence Corresponding Test Page
  • 30. Confluence Corresponding Test Page
  • 31. Confluence Corresponding Test Page
  • 32. Confluence Corresponding Test Page
  • 33. Confluence Corresponding Test Page
  • 34. Confluence Corresponding Test Page
  • 35. Confluence Corresponding Test Page
  • 36. Confluence Corresponding Test Page
  • 37. Confluence Corresponding Test Page
  • 38. Confluence Corresponding Test Page
  • 39. Confluence Corresponding Test Page
  • 40. Confluence Corresponding Test Page
  • 41. SCHEDULE TO AVOID CONFLICTS BUILD OUT GOOD CANDIDATES TRIAGE BASED ON POTENTIAL IMPACT COLLECT GREAT IDEAS Our Process ANALYZE RESULTS RUN EXPERIMENTS
  • 42. JIRA Prioritized Backlog
  • 43. SCHEDULE TO AVOID CONFLICTS BUILD OUT GOOD CANDIDATES TRIAGE BASED ON POTENTIAL IMPACT COLLECT GREAT IDEAS Our Process ANALYZE RESULTS RUN EXPERIMENTS
  • 44. Optimizely Start Experiment
  • 45. Optimizely Project Code
  • 46. Magnolia CMS Include Optimizely Checkbox
  • 47. Magnolia CMS Include Optimizely Checkbox
  • 48. Confluence Corresponding Test Page
  • 49. SCHEDULE TO AVOID CONFLICTS BUILD OUT GOOD CANDIDATES TRIAGE BASED ON POTENTIAL IMPACT COLLECT GREAT IDEAS Our Process ANALYZE RESULTS RUN EXPERIMENTS
  • 50. Optimizely Results
  • 51. What are you are trying to improve?
  • 52. Blimp Dashboard
  • 53. Optimizely Results
  • 54. Confluence Corresponding Test Page
  • 55. Confluence Results Archive
  • 56. Ideas are everywhere Everything aecting conversion is tested No trac wasted Be aggressive Regular cadence Never run a test that you wouldnt want to win Our Test Philosophy
  • 57. Ideas are everywhere collect as many as you can, prioritize based on impact. Use ticket number everywhere so everyone is referring to the same test. Involve stakeholders and developers and get approvals. Key Take Aways
  • 58. Thank you! MARK HALVORSON HEAD OF INTERACTIVE ATLASSIAN SOFTWARE @halv0112