going beyond ab testing - unbounce · going beyond ab testing . how to connect customers to...

Going Beyond AB Testing

How to Connect Customers to

Experiences

1. Decision Rules as a Framework

2. Picking Rules -> Optimization

3. Multi-Armed Bandit

twitter: @mgershoff

Blog: www.conductrics.com/blog

Overview

twitter: @mgershoff

Blog: www.conductrics.com/blog

Bandity Tools

Decision Rules

Decision Rule

IF [THIS]

Decision Rule

IF [THIS]

THEN [THAT]

A Logic that links

Observations to Actions

Why Decision Rules?

Decision Rule: General Form

Conductrics twitter: @mgershoff Blog: www.conductrics.com/blog

If [Observed Data]


If [Observed Data]

Then [Causal Action]

Decision Rule: General Form


If [Anyone]

Decision Rule: Simple Test


If [Anyone]

Then [A or B]

Decision Rule: Simple Test


If [Unbounce User]

Decision Rule: Targeted Test


If [Unbounce User]

Then [ or ]

Decision Rule: Targeted Test


Two Types of Data

• Day Part

• Weekend/Weekday

• Mobile/DeskTop

• Browser Type

• User Age

• Geo/Census

• Weather

• Tenure/RFM Score

Observed Data

• The Button – Ha!

• Price

• Sales Offers

• Shipping Type

• Layout/UX

• Which Products

• Suggested

Quantities

Causal Action

Learning Efficiently

+

Apply ‘Best’ Rules

Optimization is Balance

How do we pick the rules?

But …

If THIS Then THAT

If THIS Then THAT

If THIS Then THAT If THIS Then THAT

If THIS Then THAT

If THIS Then THAT

If THIS Then THAT

If THIS Then THAT

If THIS Then THAT If THIS Then THAT

If THIS Then THAT

If THIS Then THAT

Learn From Domain Experts …

* Image Source http://en.wikipedia.org/wiki/Anna_Wintour

…and from Experiments

* Image Source http://en.wikipedia.org/wiki/Long_Ashton_Research_Station

22 [email protected]; www.conductrics.com

Multi-Armed Bandit Experiment

Which is Better to Experiment?

Vs

"DevonLarratt-TarasIvakin" by Above99 - Own work. Licensed under CC BY 3.0 via Wikimedia Commons - http://commons.wikimedia.org/wiki/File:DevonLarratt-TarasIvakin.jpg#/media/File:DevonLarratt-TarasIvakin.jpg

Source: http://www.startrek.com/article/ten-for-ward-10-favorite-kirk-fight-scenes

Experimentation Review


The A/B

TEST

Anyone

A

B

Simple AB: Data Collection

Anyone

A

B

Simple AB: Evaluation Phase

1) Calculate Average/Means of A & B

2) Calculate Spread/Variance in A & B

+

Mean(B) – Mean(A) t=

4) Use evidence to make a selection

3)


(Or favorite Test)

Simple AB: Evaluation Phase

B A

29

Simple AB: Apply Rule

If [Anyone]

Offer B Then

MAB

How Does it Work?


Search in Most

Promising Areas


A measure of Uncertainty

Mean - 2*Std +2*Std


Recall Confidence Interval

Mean +Bonus

Score each option using the upper

portion of the interval as a Bonus


Upper Confidence as Bonus

0

Conversion Rate

5% 10%

A

B

C

1) Use upper portion of CI as ‘Bonus’

Select A

2) Select based on highest ‘Score’



0 5% 10%

A

B

C

1) Selecting Action ‘A’ reduces uncertainty

bonus (because more data)

Select C

2) Action ‘C’ now has highest score



Conversion Rate

When MAB?

• Headlines

• Flash Sales

Adaptive/Bandit AB Test


Is Learning Perishable?

Yes! Not Really

• Site Redesign

• Many Options

• Many Tests

• Personalized

• Top ‘k of M


Complex Problem?

Yes! Not Really

• Few Options

• Few Tests

• No/Simple Targeting

• Single Option

Pfizer Use Case The ‘IF‘ Part

• Three Different User Segments

• Wyeth Pharma Search Segment

• King Pharma Search Segment

• All others Segment

Conductrics Confidential

Pfizer Use Case

The ‘Then’ Part

• Best Content Topic for each Segment

• Best Navigation UI

The ‘IF‘ Part


• Wyeth Pharam Search Segment




Pfizer Use Case

The ‘Then’ Part

• Best Content Topic for each Segment

• Best Navigation UI

The ‘IF‘ Part


• Wyeth Pharam Search Segment



The ‘How’ Part

• Test Bandit Approach


Pfizer Use Case


Topics

1)Popular: Default 2) Wyeth Related 3) King Related


Investor Navigation

1) NavA: Default

2) NavB: Box

3) NavC: Tab

Create Agent

Create Segments

Wyeth Interested

King Interested

Top Option 8 times more often than the default


Results: Wyeth Segment

0%

2%

4%

6%

8%

10%

12%

Tab Box Tab Box Tab Box Tab Box

Wyeth King

Bandit Random

Bandit Random


Confidence Intervals MAB & AB C

on

vers

ion

Rate

0%

2%

4%

6%

8%

10%

12%


Wyeth King

Bandit Random

Bandit Random


Co

nve

rsio

n R

ate

High Value Option-> Smaller CIs

0%

2%

4%

6%

8%

10%

12%


Wyeth King

Bandit Random

Bandit Random


Low Value -> Higher Uncertainty C

on

vers

ion

Rate

All Others

Wyeth Searchers

King Searchers

Content Options

Navigation Options

• For the segments - the content selection had a greater impact

• For ‘all others’ – the Navigation selection had the most impact


Nav or Topics: Impact

1. Automation

2. Changing World/Perishable

3. Complexity / Scale


When to Use Bandits

See Russell & Norvig ‘s Artificial Intelligence: A Modern Approach

Optimization

Just 5 Parts

Goal

Measureable Goals

Ability to ‘Sense’ the Environment

Ability to Act/Execute

A B

Observe Outcomes

Play ‘Better’ Options More Often

If Then Receive

Learn Decision Logic

Wake up. We are Done!

Twitter:mgershoff Email:[email protected]

Image source: http://www.washingtonpost.com/blogs/london-2012-olympics/wp/tag/soccer/ (Andrew Medichini — Associated Press)

going beyond ab testing - unbounce · going beyond ab testing . how to connect customers to...

Documents