A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Download A/B Testing - How to Break Things and Fail Fast...without Breaking Things

Post on 10-May-2015

240 views

Category:

Software

0 download

DESCRIPTION

In this talk Mark Halvorson will explore how Atlassian, the creator of JIRA and Confluence, has used Magnolia CMS and other tools to operationalize A/B testing to continually optimize Atlassian's website.

TRANSCRIPT

  • 1.MARK HALVORSON HEAD OF INTERACTIVE ATLASSIAN SOFTWARE @halv0112 A/B Testing HOW TO BREAK THINGS AND FAIL FAST WITHOUT BREAKING THINGS

2. About Atlassian Products include: JIRA, Conuence, Bitbucket, HipChat Founded in 2002 Over 35,000 Customers Over 700 employees globally No sales people ! ! ! We h e l p g r e a t t e a m s b u i l d b e t t e r s o f t w a r e , t o g e t h e r. 3. Think big, act small, fail fast, learn rapidly. "LEAN SOFTWARE DEVELOPMENT BY MARY AND TOM POPPENDIECK (2003) 4. Atlassian A/B Testing is cross functional Email Product Management Customer Platform Customer Advocates Internal Systems ! ! ! C e n t r a l G r o w t h H a c k i n g Te a m t o s u p p o r t : Leads Product Marketing Support Bitbucket HipChat 5. SCHEDULE TO AVOID CONFLICTS BUILD OUT GOOD CANDIDATES TRIAGE BASED ON POTENTIAL IMPACT COLLECT GREAT IDEAS Our Process ANALYZE RESULTS RUN EXPERIMENTS 6. SCHEDULE TO AVOID CONFLICTS BUILD OUT GOOD CANDIDATES TRIAGE BASED ON POTENTIAL IMPACT COLLECT GREAT IDEAS Our Process ANALYZE RESULTS RUN EXPERIMENTS 7. Where do experiment ideas come from? 8. Anywhere. 9. What are you are trying to improve? 10. Blimp Dashboard 11. JIRA Ticket 12. Pebbles Change in copy or CTA presentation Rocks New page (e.g. no Dev Tools family page) Boulders Simplied order form. ! ! ! Types of tests 13. Examples 14. Reduce Friction 15. Increase Visibility 16. Test drive campaigns 17. Simplify 18. SCHEDULE TO AVOID CONFLICTS BUILD OUT GOOD CANDIDATES TRIAGE BASED ON POTENTIAL IMPACT COLLECT GREAT IDEAS Our Process ANALYZE RESULTS RUN EXPERIMENTS 19. What are you are trying to improve? 20. Blimp Dashboard 21. JIRA Prioritized Backlog 22. JIRA Ticket 23. JIRA Ticket 24. Experiment Illuminati 25. SCHEDULE TO AVOID CONFLICTS BUILD OUT GOOD CANDIDATES TRIAGE BASED ON POTENTIAL IMPACT COLLECT GREAT IDEAS Our Process ANALYZE RESULTS RUN EXPERIMENTS 26. Optimizely Build 27. Optimizely Build 28. Prepare for results. 29. Confluence Corresponding Test Page 30. Confluence Corresponding Test Page 31. Confluence Corresponding Test Page 32. Confluence Corresponding Test Page 33. Confluence Corresponding Test Page 34. Confluence Corresponding Test Page 35. Confluence Corresponding Test Page 36. Confluence Corresponding Test Page 37. Confluence Corresponding Test Page 38. Confluence Corresponding Test Page 39. Confluence Corresponding Test Page 40. Confluence Corresponding Test Page 41. SCHEDULE TO AVOID CONFLICTS BUILD OUT GOOD CANDIDATES TRIAGE BASED ON POTENTIAL IMPACT COLLECT GREAT IDEAS Our Process ANALYZE RESULTS RUN EXPERIMENTS 42. JIRA Prioritized Backlog 43. SCHEDULE TO AVOID CONFLICTS BUILD OUT GOOD CANDIDATES TRIAGE BASED ON POTENTIAL IMPACT COLLECT GREAT IDEAS Our Process ANALYZE RESULTS RUN EXPERIMENTS 44. Optimizely Start Experiment 45. Optimizely Project Code 46. Magnolia CMS Include Optimizely Checkbox 47. Magnolia CMS Include Optimizely Checkbox 48. Confluence Corresponding Test Page 49. SCHEDULE TO AVOID CONFLICTS BUILD OUT GOOD CANDIDATES TRIAGE BASED ON POTENTIAL IMPACT COLLECT GREAT IDEAS Our Process ANALYZE RESULTS RUN EXPERIMENTS 50. Optimizely Results 51. What are you are trying to improve? 52. Blimp Dashboard 53. Optimizely Results 54. Confluence Corresponding Test Page 55. Confluence Results Archive 56. Ideas are everywhere Everything aecting conversion is tested No trac wasted Be aggressive Regular cadence Never run a test that you wouldnt want to win Our Test Philosophy 57. Ideas are everywhere collect as many as you can, prioritize based on impact. Use ticket number everywhere so everyone is referring to the same test. Involve stakeholders and developers and get approvals. Key Take Aways 58. Thank you! MARK HALVORSON HEAD OF INTERACTIVE ATLASSIAN SOFTWARE @halv0112