Matt StodolnicVP Marketing & Alliances

Pantheon

Brad TaylorSenior Frontend Engineer

Optimizely

Matthew AuerbachDeveloper Evangelist

Optimizely

➔

➔

➔

➔

➔

Importance of Experimentation

●

●

○

●

What is A/B Testing?

Comparing two variations of a webpage to see which performs better

Hypothesis: Attracting more attention to the SHOP NOW button will increase

impressions to the item catalog

A B

Business Objective: Reduce cost of free shipping program and deliver better online shopping experience

Goal: Encourage visitors to purchase only one instead of multiple sizes

Experiment: Target a helpful modal to ‘size-conscious’ shoppers to assist them with finding the correct fit

Impact: 80% decrease in return rate for‘size-conscious’ shoppers

Case Study Brooks

VARIATION

80%Fewer returns

Visual Editor

The Optimizely Snippet

https://help.optimizely.com/Set_Up_Optimizely/Implement_the_Optimizely_snippet

Each project has its own project code, called the snippet.

Optimizely snippet contents:

• Active Experiments / Campaigns*• Draft Experiments / Campaigns*• Project Metrics• Project Audiences• Extensions*• Optimizely execution and tracking logic

* Optimizely can be configured to either remove these from the snippet or load them asynchronously.

Drupal ModuleMakes it easy to define which pages to load the Optimizely JS, no native editor functionality.

WordPress Install JS snippet directly in the head module

Easy Access to A/B Testing

● Secure the necessary tools

● Observe user behavior

● Develop a hypothesis

● Create tests for your hypothesis

● Analyze results

● Google Analytics or other analytics tool (make sure tools are working).

● Optimizely or other easy to use, easy to setup (SaaS/JS include) testing tool.

● Crazy Egg or other user behavior / click tracking tool

Google Behavior Flow.Where are people going on our website?Click-path analysis (by URL)

Crazy EggWhat are people clicking on our website?More visual click-path analysis (by clicks)

More clicks, please.

More clicks, please.

● New Messaging will connect with our visitors and more our our visitors will navigate to the tools pages of our website.

● Sign-Ups will increase because we are our value propositions are more aligned to our audience.

● A| B Testing Almost anything on your website that affects visitor behavior can be A/B tested.

● Testing Complete Pages like Pantheon did for this test requires more planning for redirection of traffic and measuring multiple changes at once.

50% - CONTROL 50% - VARIATION

● Engagement - % of visitors who clicked on any part of the experiment page

● Traffic to new user-specific landing pages - % of visitors who visited any of the three new landing pages

● Developer Registration - % of visitors who registered for a free developer account

A B C

● New Homepage - A new homepage is both an A/B test and a multivariate test.

● Just Start Testing - Agree with testing purists and move on. Improve the site, create new tests. Improve the site, create new tests.

• Improve the user experience

• Verify new product features

• Drive specific KPIs

• Test messaging

• Measure algorithm performance

• Build a culture of experimentation

Why is A/B testing important?

Working with your organization

Enable Rapid Experimentation by Everyone

Built for business users

See results in real-time Act with confidence

and developers

Always-onexperimentation

Experiment across your organization and business objectives

Marketing

Data Science

Engineering

Operations

Product

Grow revenueReduce costsBuild brand

MaximizeProfits

Experiment to roll out new features

Experiment to optimize your site’s functionality

Experiment with user experience to drive engagement

Experiment to better understand recommendation algorithms

Experiment for lead generation and conversion rate

Why stats engine

1920sData is expensive

Data is slow

Practitioners are trained experts

2017

Data is cheap

Data is real-time

Everyone is a practitioner

Back then, stats was hard!

Before Stats Engine: Lots of work to get valid results

To get valid results for a test, you had to follow some rules:

● Set a sample size in advance● No peeking● Can’t test too many goals and

variations

After Stats Engine: Always valid results to make decisions in real-time

Sequential testingLook at results in real time

False discovery rate controlTest as many goals and variations as needed

= Always valid results with Stats Engine

Optimizely platform from a dev perspective

Experiment SDKs

Developer Tools

REST APIBehavior API

Event API

Data ExportExtensions

Web Tools

Customer Profiles

Optimizely X: Operationalize Experimentation Across the Organization

Enable Rapid Experimentation by Everyone

Experiment Everywhere on Any Channel and Device

Deliver High-Performing Personalized Experiences

Optimize Anything that Matters to your Business

Extensions

• Reusable templates for making changes in the visual editor.

• A developer can build a extension with HTML, CSS, and JavaScript

• Customizable fields allow any user to alert the Extension

• Pop-up message, banner, countdown timer, etc…

• Sample Library

Extensions

Get started with experimentation today!

Documentation and developer information: developers.optimizely.com

Free trial: optimizely.com/free-trial