optimization summit 2013 - meclabs...as well as an optimization roadmap to operationalize big data...

Radical Transformations How American Express reimagined the customer experience with a data-driven approach to optimization

Sabrina PasiniSenior Marketing ManagerAmerican Express

Selena BlueManager of Editorial ContentMECLABS

Session speaker

Headshot

@s_e_pasini

Sabrina Pasini

Senior Marketing Manager

American Express

As Senior Manager of Digital Acquisition at American Express, Sabrina manages the delivery of innovative and high-profile Web projects that maximize the performance and user experience of the American Express Consumer Card website. This has included overseeing a new site redesign, ongoing test and learn campaigns, as well as an optimization roadmap to operationalize big data for real-time targeting. Pasini has been with American Express for more than five years and has held a variety of key roles in the digital acquisition group.

About American Express

107.2million

cards-in-force

FORTUNE 500 RANKING: 90

employees worldwide62,848

$33billon inannual

revenue

countries130+

More than 20 credit cards

The Challenge: Transforming digital experiences

Rethink the digital

experience

Ditch educated

guesswork for testing

Implement strategic

optimization roadmap

Deliver targeted and personalized experiences

Challenge: Transforming digital experiences

Rethink the digital

experience

Implement strategic

optimization roadmap

Ditch educated

guesswork for testing

Deliver targeted and personalized experiences

How do you accomplish this?

Testing progressionExpert Judgment A/B Testing Multivariate Testing

Segment-based Multivariate Testing

Step 1. Expert judgment

Step 1. Expert judgment

Testing progressionExpert Judgment A/B Testing Multivariate Testing

Segment-based Multivariate Testing

Step 2. A/B testing

11

Background: Test the existing Card Member-focused homepage with treatments featuring three and four cards in the hero section.

Objective: To increase card application conversion rate.

Primary Research Question: Which page emphasis – specific popular cards or balanced selection across portfolios – will result in higher conversion rates?

Test Design: 100% redirect from Current Cards Landing to 3 Cards Landing – 4% random, 5% default and 91% targeted

Experiment Research Notes:

Step 2. A/B testingControl

Step 2. A/B testingControl Treatment 1

Step 2. A/B testingControl Treatment 1 Treatment 2

Step 2. A/B testingControl Treatment 1 Treatment 2

Step 2. A/B testingControl Treatment 1 Treatment 2

Resulted in No Clear Winner

Testing progressionExpert Judgment A/B Testing Multivariate Testing

Segment-based Multivariate Testing

Step 3. Multivariate testing

18

Background: Test for further improvement on Treatment 1 from the previous A/B test.

Objective: To increase card application conversion rate.

Primary Research Question: Will optimizing each of the key elements on the page drive engagement and increase conversion rate?

Test Design: Full-factorial MVT with six elements for a total of 256 variations. With business exclusion applied, a total of 144 experiences are tested.

Experiment Research Notes:

Step 3. Multivariate testingVariable #1. Color scheme

Step 3. Multivariate testingVariable #2. Page navigation

Step 3. Multivariate testingVariable #3. Calls-to-action

Step 3. Multivariate testingVariable #4. Comparison tab

Step 3. Multivariate testingVariable #5. Lower tool section

Step 3. Multivariate testingVariable #6. Tool in tab

Step 3. Multivariate testing

Which combination won?

Step 3. Multivariate testingWinning CombinationControl

Step 3. Multivariate testingWinning Combination

Increase in Conversion Rate29%

Control

Ranking of MVT treatment results

Control Winning Combination

1

?

144

Ranking of MVT treatment results

144 119

Control

1

Winning Combination

The next frontier

One-size-fits-all

interactions

Experiences customized to

microsegments

Testing progressionExpert Judgment A/B Testing Multivariate Testing

Segment-based Multivariate Testing

Balancing Act: Business priority vs. customer needs

Higher Value

LowerCombined

Value

Segmentation process

3rd

Party

Model Scores

Offsite Media

Prospect Data

Customer Data

Site Activity

Step 1. Track and Identify

Prospect Traits

COOKIE:1048379205

Segmentation process

3rd

Party

Model Scores

Offsite Media

Prospect Data

Customer Data

Site Activity

Step 1. Track and Identify

Prospect Traits

COOKIE:1048379205

Step 2. Combine Traits into Segments

Segmentation process

3rd

Party

Model Scores

Offsite Media

Prospect Data

Customer Data

Site Activity

Step 1. Track and Identify

Prospect Traits

COOKIE:1048379205

Step 3. Displays Custom

Content in Real Time

Step 2. Combine Traits into Segments

Step 4. Segment-based multivariate testing

36

Background: Test personalized homepages for a customer-centric experience.

Objective: To increase response and conversion rate.

Primary Research Question: Will targeting cards to profiles that may have an affinity for these products increase clickthrough and conversion rates?

Test Design: Target created four campaigns: 100% Redirect from Current Cards Landing to 3 Cards Landing – 4% random, 5% default and 91% targeted.

Experiment Research Notes:

Step 4. Segment-based multivariate testing

Match “Member Story” to featured

product

Display next-highest valued

products

Feature the winning product

Step 4. Segment-based multivariate testing

Control Treatment

Step 4. Segment-based multivariate testing

Control Treatment

Increase in Conversion Rate28%

Segmentation example

Step 1. Cookie assigned

Segmentation example

Step 1. Cookie assigned

Assigned Cookie:8762094795

Prospect:Robert

Segmentation example

Step 1. Cookie assigned

Step 2. Traits tracked before

Card homepage arrival

Segmentation example

Step 1. Cookie assigned

Step 2. Traits tracked before

Card homepage arrival

Interest: Travel

• 3rd Party: Travel

• Geo: Atlanta

• DFA: Delta 300x250

• DFA: Delta 160x600

Segmentation example

Step 1. Cookie assigned

Step 2. Traits tracked before

Card homepage arrival

Step 3. Customized Card

homepage presented

Interest: Travel

Interest: Travel

Segmentation example

Step 1. Cookie assigned

Step 2. Traits tracked before

Card homepage arrival

Step 3. Customized Card

homepage presented

Segmentation example

Step 1. Cookie assigned

Step 2. Traits tracked before

Card homepage arrival

Step 3. Customized Card

homepage presented

Step 4. Prospect continuesto search website

Interest: Travel

Interest: Travel

Segmentation example

Step 1. Cookie assigned

Step 2. Traits tracked before

Card homepage arrival

Step 3. Customized Card

homepage presented

Step 4. Prospect continuesto search website

Segmentation example

Step 1. Cookie assigned

Step 2. Traits tracked before

Card homepage arrival

Step 3. Customized Card

homepage presented

Step 4. Prospect continuesto search website

Step 5. Newly acquired traits assigned to prospect

Interest: Travel

Interest: Travel

Segmentation example

Step 1. Cookie assigned

Step 2. Traits tracked before

Card homepage arrival

Step 3. Customized Card

homepage presented

Step 4. Prospect continuesto search website

Step 5. Newly acquired traits assigned to prospect

Intent: Cashback

• View All Cards: Cashback

• BCE Learn More Page

• BCE Credit QuickInfo

• View All Cards: Credit

Segmentation example

Step 1. Cookie assigned

Step 2. Traits tracked before

Cards homepage arrival

Step 3. Customized Cards

homepage presented

Step 4. Prospect continuesto search website

Step 5. Newly acquired traits assigned to prospect

Step 6. Updated Cards

homepage presented

Interest: Travel

Intent: Cashback

Intent: Cashback

Interest: Travel

Segmentation example

Step 1. Cookie assigned

Step 2. Traits tracked before

Cards homepage arrival

Step 3. Customized Cards

homepage presented

Step 4. Prospect continuesto search website

Step 5. Newly acquired traits assigned to prospect

Step 6. Updated Cards

homepage presented

Segmentation example

Step 1. Cookie assigned

Step 2. Traits tracked before

Cards homepage arrival

Step 3. Customized Cards

homepage presented

Step 4. Prospect continuesto search website

Step 5. Newly acquired traits assigned to prospect

Step 6. Updated Cards

homepage presented

Interest: Travel

Intent: Cashback

The new optimization roadmap

LayoutA/B Test

Functionality/UXMultivariate Testing

ProductA/B/n Targeting

SegmentationCluster Analysis

ExperienceSegmentation & Personalization II

PositioningSegmentation & Personalization I

New

Acc

ou

nts

Time

Relevant Product

Incentive Strategy

Messaging/ Customer Journey

Enabling personalized experiences using big data

Estimated result of personalization

Increase in estimated revenue of build business acquired by 2015

50%

Overall results for optimization project

Nominated for The American Express President Award for Innovation in 2013.

Earned American Express the 2013 Internet Advertising Competition Award for Outstanding Achievement in Internet Advertising.

The new site scored first place in Forrester’s 2013 US Credit Card Online Sales Rankings.

Key takeaways

Put customers first

Let the data do the work for you

Never stop improving

Thank you

Sabrina Pasini

Senior Marketing Manager

American Express

@s_e_pasini

Selena Blue

Manager of Editorial Content

MECLABS

@SelenaLBlue

Headshot