Ben Day

Group Marketing Manager

Microsoft

Case Study: How Microsoft Uses Technology to Drive 1:1 Real-Time Content and Offer Optimization in Email to 2 Billion Consumers

• Microsoft Online Division: 2 Billion MSN, Bing and Windows Live consumers in 122 countries

• Large-Scale Email:

70,378 cells (+61% YoY)

3,410 campaigns (+35% YoY)

Total email volume of ~10B

• Marketing Operations: 20+ folks handling email and display marketing, workflow automation and email product management

Case Study Background

2

The Challenge: Making Email Marketing Valuable for Your Customer

What if customers could receive the best email content at the right time?

What if emails could display the latest content, regardless of when they are opened?

What if poor performing email content could be automatically removed?

What if expired or already fulfilled content could be instantly replaced?

What if the “delete reflex” could be stopped by giving customers personalized information they care about and appreciate?

Upheaval in the Email Marketing Landscape

Traditional static email marketing shows declining success with consumers and marketers

“Non-relevant mailings continue to be the top reason why consumers opt out of email.”

D. Daniels,The ROI Of Email Relevance, September 2009

• Consumers are still opting out in large numbers due to irrelevant messages

• Blasting large numbers of generic emails also leads to delivery and reputation issues at ISPs

• Creating multiple email versions for greater personalization remains cost-prohibitive for most marketers

Email marketers list “targeting recipients with highly relevant content” as their most significant challenge.

Overcoming the challenge is important since email marketers list relevancy tactics as effective in improving results.

Before: Static, homogenized email

Static Content

Email Marketing or ESP

Multiple Template Upload

Multi-Segmentation andGlobal Rules

The current state: Email content is generated independently from end-customer preferences and remains static

Weeks/Months

After: Dynamic, intelligent email

Only Minutes/Hours

Dynamic Content

Email Marketing or ESP

Wire Frame

Header

Unsubscribe

Offer 1 Offer 2

Offer 3

Simplified Segmentation and Global Rules

Email updates instantly when opened to show most-relevant content based on unified end-customer information from all touch points

How It Works

6

2 Campaign Management

Outbound Mail Server

3

Email HTML Template sent w/o

Content

Header Slot

Unsubscribe

Dynamic

Offer 1

Dynamic

Offer 2

Dynamic

Offer3

Dynamic

Offer 4

4Customer

opens email with targeted

content.

Web/CMS Serverdelivers images, hosts link

redirects

Images, links and audience for campaign

5IA Server/

Real-time Data Minerdetermines best offers

7Images appear

in email

1

Real-Time Decisioning Framework

• Activation conditions

• Qualification conditions

Real-TimeAnalytics

• Automatic real-time targeting

• Likelihood estimation

• Collaborative filtering

Arbitration

• Directly integrate with email engine

• Real-time tracking

• Continuous feedback

Self-learning and self-adjusting

Offers

Delivery and Tracking

• Align competing interests

• Offer prioritization

Business Constraint Modeling

Real-Time Profiles

• Drive relevant offers with contextual information

o Understanding customer intent increases acceptance rates and customer satisfaction

• Avoid stale data by using real-time feeds for attributes

o Relying on last month’s batch load can cause poor targeting or out of date offers

• Leverage external market factors in real-time

o Competitor offerings, interest rates or new regulation may all affect offer success

Dynamic Profiles

Historical

Transactional

Demographic

Contextual:

Click Stream

Guided Selling Systems

Other External Sources

No data replication

Leverage Real-Time Customer Data for Rules and Analytics

Comprehensive Real-Time Analytics

• Real-Time Miner

o Dynamically build and improve predictive models based on each interaction

o Provide real-time insight on the attributes driving customer behavior

o Make effective predictions with extremely low maintenance overhead

• Real-Time Recommendation Engine

o Collaborative Filtering recommendations based on other customer behavior

o Recommend similar products, services, articles or other correlated items

REAL-TIME SELF-LEARNING ANALYTICS

Recommendation Engine

Real-Time Miner

Real-Time Self-Learning Analytics

• Allows different levels of the organization to set different levels of priority on each campaign and offer

• Prioritizes arbitration function scores according to organizational goals/policies

• Controls offer recommendations to ensure that the best offer for both customer and organization is extended

• Several out-of-the-box arbitration schemes provided

• Supports hierarchical and multi-level arbitration

• Supports extendable custom arbitration schemes

•Best Benefit

•Likelihood

•Best Expected Benefit

•Financial Scores

•By Inventory Factors

•Contractual Obligations

•Other Criteria

Arbitrationconditions

Winningoffer

Likelihoodof acceptance

Offer 140%

Offer 218%

Offer 330%

Offer 330%

Align Competing Interests With Enterprise Objectives

Offer Arbitration

Email Template andRegions

Template divided into four dynamic regions• Top• Bottom 1• Bottom 2• Bottom 3

Top Region Offers

Mobile Summer Visual Search

Three themes with two variations on each

Bottom Region Offers

Three Static Versions

1.17%

0.50% 0.27% 0.33%

1.10%

0.38% 0.17% 0.14%

0.95%

0.27% 0.10% 0.12%

Clickthrough rates on day 0 test

Best Combo Middle Combo Worst Combo

Overall Learnings

Overall Clickthrough Rate by Region

1.21%

0.52% 0.32% 0.19%

Top image always gets the most clicks. Below-the-fold images always descend left to right.

Top Region Offers by CTR

1.36% CTRServed to 67.38% of openers

1.04% CTRServed to 16.41% of openers

0.88% CTRServed to 7.22% of openers

0.78% CTRServed to 5.96% of openers

0.61% CTRServed to 2.39% of openers

0.52% CTRServed to 0.64% of openers

Bottom Region Offers

.59% CTR .51% CTR .50% CTR .35% CTR .26% CTR .26% CTR

.35% CTR .21% CTR .21% CTR .17% CTR .16% CTR .13% CTR

.28% CTR .23% CTR .14% CTR .12% CTR .11% CTR .10% CTR

Bo

tto

m 1

Bo

tto

m 2

Bo

tto

m 3

Email Advisor Learnings

Top 5 Influencing AttributesDays Since Messenger UpgradeMessenger Client Version DescriptionMessenger Month UsageMessenger Number of Buddies in Friends ListPassport Postal Code

Days Since Messenger UpgradeDays Impact14 - 30 Strong 30 - 50 Strong 50 - 70 Moderate 70 - 100 Moderate 100 – 130 Moderate 130 - 160 Moderate 230 - 260 Moderate 160 - 190 Weak 260 - 330 Weak 190 - 210 Weak 210 - 230 Insignificant 330 - 430 Insignificant 430 - 890 Weak negative 890 - 40330 Moderate negative 40330 - 40340 Strong negative 40340 - 40350 Very strong negative 40350 - 40354 Very strong negative

In IA, all attributes and attribute values are ranked according to response rates, and the real-time decision engine uses these values to determine which offer to display to each opener.

Email Advisor Learnings

Top 5 Influencing AttributesAgeHotmail Contacts Month Visit CountHotmail Month UsageMessenger Client Version DescriptionPassport Postal Code

Age68 - 109 Strong 63 - 67 Strong 60 - 62 Strong 58 - 59 Moderate 55 - 57 Moderate 50 - 51 Moderate 52 - 54 Weak 47 - 49 Weak 44 - 46 Insignificant 41 - 43 Weak negative 39 - 40 Weak negative

34 - 35 Moderate negative 36 - 38 Moderate negative 32 - 33 Moderate negative 30 - 31 Strong negative 28 - 29 Strong negative 26 - 27 Strong negative 23 - 25 Strong negative 20 - 22 Strong negative 2 - 19 Very strong negative

Top 5 Influencing AttributesDays Since Messenger UpgradeMessenger Client Version DescriptionMessenger Month UsagePassport GenderPassport Postal Code

Passport GenderM Moderate m Moderate U Weak negative F Strong negative f Strong negative

Top 5 Influencing AttributesAgeCEF SegmentDays Since Messenger UpgradeMessenger Client Version DescriptionPassport Postal Code

CEF SegmentCEF_ConsistentSecondary Moderate CEF_ConsistentPrimary Moderate CEF_Occassional Moderate CEF_OneMonthSecondary Moderate CEF_OneMonthPrimary Insignificant CEF_Non-Bing Moderate negative

Sample drill-down data on bottom region offers

Email Advisor Clickthrough Lift

0%

20%

40%

60%

80%

IA Treatment Lift

Best Combo

Mid Combo

Worst Combo

Interaction Advisor achieved uplift against all scenarios

Even if, by expertise or luck, we chose all the best offers during the creative review process, IA would still provide a significant incremental return.

Treatment Sent Delivered Delivery Rate Opens Open Rate ClicksClick Rate Difference Lift

Interaction Advisor 6,180,381 6,167,481 99.79% 1,955,347 31.70% 86,817 4.44%

Best Combo 263,185 262,704 99.82% 83,534 31.80% 3,004 3.60% 0.84% 23.47%

Mid Combo 255,446 254,987 99.82% 80,831 31.70% 2,590 3.20% 1.24% 38.57%

Worst Combo 255,445 255,021 99.83% 80,644 31.62% 2,154 2.67% 1.77% 66.23%

0.00%

1.00%

2.00%

3.00%

4.00%

5.00%

Click-Through Rate

Best Combo

Mid Combo

Worst Combo

IA

Email Channel Content Decisioning and Delivery

1. I have a customer…

4. Returns offer

2. Upon opening an

email…

5. Continuous learning

and update models

3. Begins

Processing

“Power View”

Demographics

Transaction data

Contextual

Real-Time

AnalyticsBusiness Rules ArbitrationDynamic

Profiling

Campaigns activation

and qualification

Offers governance

Offers history

Automatic real-time

targeting

Likelihood estimation

Collaborative filtering

3rd party statistical

model

Aligns customer

interests and

organization objectives

Support several out-of-

box and custom

arbitration schemes

Email Wire Frame

Unsubscribe

Offer 1 Offer 2

Offer 3

Prioritized / Personalized Content, Message, Offer

Key Advantages of Dynamic Real-Time Messaging

• Can help you “break from the pack” and differentiate your company

• Relevant, personalized content shows your customers that you understand

what’s important to them

• Personalized outbound content is seeing measurable lift as compared to

static content

• You can impact all interactions – regardless of touch point

• Allows easy integration between outbound and inbound marketing

• Can help you deliver the “Next Best Action” based on real-time analytics

• You can leverage learnings in all customer channels

• Data can help you refine your Outbound Marketing emails

Credits

• The Microsoft team:

o Kerry Godes, Group Marketing Manager, Online Division

• Consultants:

o Todd Gile, Regional Sales Director, Infor CRM

• The Platform: Infor