optimization summit 2013 - meclabs...as well as an optimization roadmap to operationalize big data...
TRANSCRIPT
Radical Transformations How American Express reimagined the customer experience with a data-driven approach to optimization
Sabrina PasiniSenior Marketing ManagerAmerican Express
Selena BlueManager of Editorial ContentMECLABS
Session speaker
Headshot
@s_e_pasini
Sabrina Pasini
Senior Marketing Manager
American Express
As Senior Manager of Digital Acquisition at American Express, Sabrina manages the delivery of innovative and high-profile Web projects that maximize the performance and user experience of the American Express Consumer Card website. This has included overseeing a new site redesign, ongoing test and learn campaigns, as well as an optimization roadmap to operationalize big data for real-time targeting. Pasini has been with American Express for more than five years and has held a variety of key roles in the digital acquisition group.
About American Express
107.2million
cards-in-force
FORTUNE 500 RANKING: 90
employees worldwide62,848
$33billon inannual
revenue
countries130+
More than 20 credit cards
The Challenge: Transforming digital experiences
Rethink the digital
experience
Ditch educated
guesswork for testing
Implement strategic
optimization roadmap
Deliver targeted and personalized experiences
Challenge: Transforming digital experiences
Rethink the digital
experience
Implement strategic
optimization roadmap
Ditch educated
guesswork for testing
Deliver targeted and personalized experiences
How do you accomplish this?
Testing progressionExpert Judgment A/B Testing Multivariate Testing
Segment-based Multivariate Testing
Step 1. Expert judgment
Step 1. Expert judgment
Testing progressionExpert Judgment A/B Testing Multivariate Testing
Segment-based Multivariate Testing
Step 2. A/B testing
11
Background: Test the existing Card Member-focused homepage with treatments featuring three and four cards in the hero section.
Objective: To increase card application conversion rate.
Primary Research Question: Which page emphasis – specific popular cards or balanced selection across portfolios – will result in higher conversion rates?
Test Design: 100% redirect from Current Cards Landing to 3 Cards Landing – 4% random, 5% default and 91% targeted
Experiment Research Notes:
Step 2. A/B testingControl
Step 2. A/B testingControl Treatment 1
Step 2. A/B testingControl Treatment 1 Treatment 2
Step 2. A/B testingControl Treatment 1 Treatment 2
Step 2. A/B testingControl Treatment 1 Treatment 2
Resulted in No Clear Winner
Testing progressionExpert Judgment A/B Testing Multivariate Testing
Segment-based Multivariate Testing
Step 3. Multivariate testing
18
Background: Test for further improvement on Treatment 1 from the previous A/B test.
Objective: To increase card application conversion rate.
Primary Research Question: Will optimizing each of the key elements on the page drive engagement and increase conversion rate?
Test Design: Full-factorial MVT with six elements for a total of 256 variations. With business exclusion applied, a total of 144 experiences are tested.
Experiment Research Notes:
Step 3. Multivariate testingVariable #1. Color scheme
Step 3. Multivariate testingVariable #2. Page navigation
Step 3. Multivariate testingVariable #3. Calls-to-action
Step 3. Multivariate testingVariable #4. Comparison tab
Step 3. Multivariate testingVariable #5. Lower tool section
Step 3. Multivariate testingVariable #6. Tool in tab
Step 3. Multivariate testing
Which combination won?
Step 3. Multivariate testingWinning CombinationControl
Step 3. Multivariate testingWinning Combination
Increase in Conversion Rate29%
Control
Ranking of MVT treatment results
Control Winning Combination
1
?
144
Ranking of MVT treatment results
144 119
Control
1
Winning Combination
The next frontier
One-size-fits-all
interactions
Experiences customized to
microsegments
Testing progressionExpert Judgment A/B Testing Multivariate Testing
Segment-based Multivariate Testing
Balancing Act: Business priority vs. customer needs
Higher Value
LowerCombined
Value
Segmentation process
3rd
Party
Model Scores
Offsite Media
Prospect Data
Customer Data
Site Activity
Step 1. Track and Identify
Prospect Traits
COOKIE:1048379205
Segmentation process
3rd
Party
Model Scores
Offsite Media
Prospect Data
Customer Data
Site Activity
Step 1. Track and Identify
Prospect Traits
COOKIE:1048379205
Step 2. Combine Traits into Segments
Segmentation process
3rd
Party
Model Scores
Offsite Media
Prospect Data
Customer Data
Site Activity
Step 1. Track and Identify
Prospect Traits
COOKIE:1048379205
Step 3. Displays Custom
Content in Real Time
Step 2. Combine Traits into Segments
Step 4. Segment-based multivariate testing
36
Background: Test personalized homepages for a customer-centric experience.
Objective: To increase response and conversion rate.
Primary Research Question: Will targeting cards to profiles that may have an affinity for these products increase clickthrough and conversion rates?
Test Design: Target created four campaigns: 100% Redirect from Current Cards Landing to 3 Cards Landing – 4% random, 5% default and 91% targeted.
Experiment Research Notes:
Step 4. Segment-based multivariate testing
Match “Member Story” to featured
product
Display next-highest valued
products
Feature the winning product
Step 4. Segment-based multivariate testing
Control Treatment
Step 4. Segment-based multivariate testing
Control Treatment
Increase in Conversion Rate28%
Segmentation example
Step 1. Cookie assigned
Segmentation example
Step 1. Cookie assigned
Assigned Cookie:8762094795
Prospect:Robert
Segmentation example
Step 1. Cookie assigned
Step 2. Traits tracked before
Card homepage arrival
Segmentation example
Step 1. Cookie assigned
Step 2. Traits tracked before
Card homepage arrival
Interest: Travel
• 3rd Party: Travel
• Geo: Atlanta
• DFA: Delta 300x250
• DFA: Delta 160x600
Segmentation example
Step 1. Cookie assigned
Step 2. Traits tracked before
Card homepage arrival
Step 3. Customized Card
homepage presented
Interest: Travel
Interest: Travel
Segmentation example
Step 1. Cookie assigned
Step 2. Traits tracked before
Card homepage arrival
Step 3. Customized Card
homepage presented
Segmentation example
Step 1. Cookie assigned
Step 2. Traits tracked before
Card homepage arrival
Step 3. Customized Card
homepage presented
Step 4. Prospect continuesto search website
Interest: Travel
Interest: Travel
Segmentation example
Step 1. Cookie assigned
Step 2. Traits tracked before
Card homepage arrival
Step 3. Customized Card
homepage presented
Step 4. Prospect continuesto search website
Segmentation example
Step 1. Cookie assigned
Step 2. Traits tracked before
Card homepage arrival
Step 3. Customized Card
homepage presented
Step 4. Prospect continuesto search website
Step 5. Newly acquired traits assigned to prospect
Interest: Travel
Interest: Travel
Segmentation example
Step 1. Cookie assigned
Step 2. Traits tracked before
Card homepage arrival
Step 3. Customized Card
homepage presented
Step 4. Prospect continuesto search website
Step 5. Newly acquired traits assigned to prospect
Intent: Cashback
• View All Cards: Cashback
• BCE Learn More Page
• BCE Credit QuickInfo
• View All Cards: Credit
Segmentation example
Step 1. Cookie assigned
Step 2. Traits tracked before
Cards homepage arrival
Step 3. Customized Cards
homepage presented
Step 4. Prospect continuesto search website
Step 5. Newly acquired traits assigned to prospect
Step 6. Updated Cards
homepage presented
Interest: Travel
Intent: Cashback
Intent: Cashback
Interest: Travel
Segmentation example
Step 1. Cookie assigned
Step 2. Traits tracked before
Cards homepage arrival
Step 3. Customized Cards
homepage presented
Step 4. Prospect continuesto search website
Step 5. Newly acquired traits assigned to prospect
Step 6. Updated Cards
homepage presented
Segmentation example
Step 1. Cookie assigned
Step 2. Traits tracked before
Cards homepage arrival
Step 3. Customized Cards
homepage presented
Step 4. Prospect continuesto search website
Step 5. Newly acquired traits assigned to prospect
Step 6. Updated Cards
homepage presented
Interest: Travel
Intent: Cashback
The new optimization roadmap
LayoutA/B Test
Functionality/UXMultivariate Testing
ProductA/B/n Targeting
SegmentationCluster Analysis
ExperienceSegmentation & Personalization II
PositioningSegmentation & Personalization I
New
Acc
ou
nts
Time
Relevant Product
Incentive Strategy
Messaging/ Customer Journey
Enabling personalized experiences using big data
Estimated result of personalization
Increase in estimated revenue of build business acquired by 2015
50%
Overall results for optimization project
Nominated for The American Express President Award for Innovation in 2013.
Earned American Express the 2013 Internet Advertising Competition Award for Outstanding Achievement in Internet Advertising.
The new site scored first place in Forrester’s 2013 US Credit Card Online Sales Rankings.
Key takeaways
Put customers first
Let the data do the work for you
Never stop improving
Thank you
Sabrina Pasini
Senior Marketing Manager
American Express
@s_e_pasini
Selena Blue
Manager of Editorial Content
MECLABS
@SelenaLBlue
Headshot