a/b optimizely technical details

A/B Testingwhat’s that and how it works

Konrad Synoradzki Allegro Group 2013

What is A/B testing?

A/B testing is comparing two versions of a web

page to see which one performs better.


web-page visitors


slice visitors


Then compare two web pages by showing the two variants (let's call them A and B) to similar visitors at the same time.


example


What can you test?• Media mentions

• Awards and badges

• Banners efficiency

• Rotation speed

• New version of page

• Removing ads


• Headlines

• Sub headlines

• Paragraph Text

• Testimonials

• Call to Action text

• Call to Action Button

• Links

• Images

• Content above or below the fold

• Social proof

Types of tests


A/B testingBest used to measure the

impact of a two to four

variables on interactions with

the page.

Examples:

• new layout

• Labels on button


Multivariate Testing• compares a higher

number of variables,

• reveals more information about how these variables interact with one another.

• traffic to a page is split between different versions of the design.

Examples

• Test headline, buttoncolor, button text and image on landing page


Multi-Page Testing• A/B test on multi-page

Example:

• Colors on primary

action in process

• Badges on promo items


Getting StartedWhat is and how to use optimizely?


Getting Started


Step 0: Implement the snippet

Getting Started


Step 1: Enter your URL on Optimizely.com

Getting Started


Step 2: Add page variations

Getting Started


Step 3: Set up goals for tracking

Getting Started


Step 4: Target and allocate

Report page


Short moviehttps://fast.wistia.net/embed/iframe/gxttknv8aa?auto

Play=true&controlsVisibleOnLoad=true&version=v1&vo

lumeControl=true


https://fast.wistia.net/embed/iframe/gxttknv8aa?autoPlay=true&controlsVisibleOnLoad=true&version=v1&volumeControl=true







Technical detailsbut how??


Organizing Experiments


User Roles


Each Project Has Its Own Project Code (Snippet)


or asynchronous version

Optimizely Snippet Contents


• Active Experiments• Draft Experiments• Project Goals• Optimizely execution and tracking logic• Optimizely jQuery Settings (optional – version 1.6.4 included by

default)

Two modes


Optimizely translates every change made in the visual editor

into a line of jQuery code, which we call Variation Code:


Optimizely Testing Process


Order of Execution


Snippet Timing Logic


Optimizely Architecture Overview


Optimizely Architecture Overview


• Google App Engine is used to store and manage your account and experiment configuration settings, including your account login and password information. Passwords are hashed and stored securely using 160 bit cryptographic hashes.

• Amazon Elastic Compute Cloud and Simple Storage Service is used to store your conversion/traffic logging data, which Optimizely uses to compute experiment results.

• Akamai is used to store and serve your Optimizely snippet JavaScript file(s). This file is publicly accessible by necessity.

Collecting data


The visitor’s browser makes an asynchronous request to https://log.optimizely.com, which logs the following information:

• The Optimizelyassigned unique user ID • IDs of the experiments and variations this visitor has seen • Your Optimizely account number • Visitor’s browser/OS version • Visitor’s IP address, with last octet removed • A string identifying this conversion goal, such as the page URL • Date and time

that’s all


question?


a/b optimizely technical details

Technology