causal attribution - proposing a better industry standard for measuring digital advertising...
DESCRIPTION
Research conducted at Collective with Jeremy Stanley, CTO, Justin Evans, Strategy Officer, and Peter Weingard, CMO, discusses the pitfalls of current digital display measurement methods, and proposes alternative measures.TRANSCRIPT
ProPosing A Better industry stAndArd MeAsure of digitAl Advertising effectivenessNew research provides a foundation for a better system of measuring the effect of
digital advertising and provides a wake-up call to marketers using existing, often
badly misleading, attribution solutions.
Billions of Advertising dollArs Are Being wAsted.
That’s because industry standard measures like
click through rate (CTR), post click and post
impression attribution are not only inherently
flawed, but are being widely manipulated by
intermediaries, and often trick marketers into
optimizing away from their best prospects.
In this study, Collective presents an alterna-
tive method for measuring digital advertising,
which uses rigorous experiments to measure the
increase in desired outcomes caused by display
advertising, verses a correlation effect measured
by existing attribution solutions.
The Causal Attribution measurement approach
outlined in this research will allow advertisers, for
the first time, to accurately evaluate how their
online advertising affects the behavior of specific
audiences and measure the value generated from
their advertising spend.
Key findings:
// Existing attribution solutions are either too
subjective, misleading, or complex to provide
a meaningful industry-standard for advertising
measurement.
// Existing attribution solutions often steal credit
from other advertising sources and misdirect mar-
keters into making poor media decisions.
// Causal Attribution measures the real impact of
a campaign by creating an experiment where the
only difference between two otherwise
Identical pools of users is that one pool (the test)
was potentially exposed to advertising, and the
other pool (the control) was not – ensuring that
any difference in performance observed on the
two pools must have been caused by the decision
to advertise to the test group.
// In multiple live campaign tests, the Causal At-
tribution approach provided an unbiased method
of linking true ROI to advertising spend, including
exposing campaigns that aren’t working.
// Because the experiment is cookie based, ROI
can be cascaded down to individual Audience
Segments, providing rich insights into the types of
users who are being influenced by the advertising.
// Beyond measuring online conversions or their
proxies, Causal Attribution can be used to mea-
sure offline conversions and brand lift.
Further uses of Causal Attribution include:
// Directly comparing the performance of
different sources of media.
// Testing multiple, potentially radically different,
creatives to identify which audiences are most
influenced by each.
// Quantifying the effect of frequency (how many
times each cookie is shown an advertisement)
on ROI.
We believe that widespread adoption of Causal
Attribution could have a tremendous positive
impact on the digital advertising industry. Proper
ROI measurement would lead to increased
digital advertising spending, higher returns for
advertisers and an incentive for publishers to
create quality, engaging content. Finally, con-
sumers would benefit from fewer pay-walls, and
more relevant advertisements that could truly
help their purchase decisions.
executivesuMMAry//
causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness
2
AlgoritHMicAttriBution //
weigHtedAttriBution //
clicK tHrougH rAte //
lAst clicKAttriBution //
lAst iMPressionAttriBution //
cAusAlAttriBution //
trA
nsP
Ar
ent
BlA
cK B
ox
MisleAding AccurAte
causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness
3
The maturation of the Web as an advertising me-
dium has spawned several attribution techniques.
These techniques attempt to leverage the Web’s
unique ability to record a recipient’s response to
ad exposure in near real time to quantify the ef-
fects of a campaign on brand value and, ultimate-
ly, sales. Advertising practitioners routinely rely on
the most commonly deployed methodologies to
make significant marketing investment decisions.
Existing attribution models, however, have severe
biases, and when optimized to (or gamed by
intermediaries to “compete” on performance),
these methodologies can erode brand value.
The research that follows demonstrates, for the
first time, how a Causal Attribution solution can
provide marketers with a true accounting of what
advertisements and audiences drive ROI.
The research will also show how the current slate
of attribution methodologies fail to provide mar-
keters with a reliable and transparent method for
understanding the true value of the media they
buy, and in some cases, produces effects counter
to the marketer’s objectives.
But first, a brief overview of the most commonly
used attribution methods:
tHe ProBleM witH current online AttriBution Models // MisleAding, eAsy to gAMe, too suBjective or too coMPlex
causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness
4
Half the money i spend on advertisingis wasted; the trouble is i don’t know which half.
–John Wanamaker
“
” {As true in digitAl in 2012 As it WAs in print in 1900}
AlgoritHMicAttriBution //(soMetiMes cAlled ‘interActive’ or, ‘MediA Mix’)
In general, attribution algorithms can be divided
into two categories - weighting schemes and
predictive models - both of which have significant
theoretical and practical weaknesses. Weighting
scheme attribution algorithms rely on human
judgment to assign weights to different types
of advertising events that occurred prior to the
desired outcome. The choice of which events to
weight and how much weight to assign to each
is subjective, even when informed by data, and
may not be any more accurate than using a single
flawed measure alone.
In response to these shortcomings, other compa-
nies have developed attribution algorithms that
use computers to assign credit rather than human
derived weights. These solutions work by assem-
bling a data set of all outcomes and all advertising
events that occurred prior to those outcomes.
Then, using a multivariate predictive model where
the dependent variable is binary (desired outcome
or not) and the independent variables are all
advertising events, these algorithms attempt to
identify which types of advertising events were
more likely to precede the desired outcome than
the undesired outcome.
These computer based attribution algorithms
(often referred to as machine learning algorithms
or predictive models) suffer from significant limita-
tions when used for attribution. First and fore-
most, they often confuse correlation with causality.
This occurs when the computer finds that certain
advertisements tend to precede the occurrence
of the desired outcome in historical data (the two
are correlated). The computer concludes, incor-
rectly, that the advertisements caused the desired
outcomes, and assigns them credit.
In fact, it is often some other unmeasured factor
that was responsible for the observed desired
outcomes.
For example, suppose that a campaign is run-
ning on two very similar sites (A and B), and the
computer is trying to determine how much credit
to assign to each site. Suppose that site A was
bought with retargeting, while site B was bought
without retargeting. The computer will see signifi-
cantly higher conversions for users reached on site
A, driven by the retargeted audience being more
likely to convert regardless of whether or not they
are advertised to. This will lead the computer to
give site A far more credit than it deserves.
Compound this simple example with the
plethora of types of audience targeting,
frequency capping, distribution channels
(display, video, social, mobile, television, etc.),
and other compounding effects (convenience,
social and repeat purchases) and it becomes
clear that these computer based attribution
algorithms are just more complex, with no real
hope of discovering the ‘truth’.
While some attribution models may improve over
traditional measures, like last-click or last-view
based attribution, their “black box” nature makes
gauging success across vendors and campaigns
challenging. Only their creators truly know how
the algorithm works, what it measures and how
it evaluates ROI. Hence, it is unlikely that a
single form of Algorithmic Attribution could ever
become an industry standard.
causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness
5
Half the money i spend on advertisingis wasted; the trouble is i don’t know which half.
–John Wanamaker”
confuses correlAtion for cAusAtion.
definitionComputer algorithm assigns credit
for outcomes by analyzing data.
ProsCan overcome some limitations
in simpler metrics.
BottoM line“Pay no attention to the man behind
the curtain.”
consOften confuses correlation
for causation.
Requires measuring all user
interactions in all channels.
No industry standard likely to
be developed.
Only programmers really know how
they work.
wHAt you getRIgHT AuDIENCE
RIgHT mEDIA
RIgHT CREATIvE
? ? ? ? ?? ? ? ? ?? ? ? ? ?
The de facto standard of web metrics, CTR
reports the cumulative number of clicks recorded
on a display ad campaign, divided by the
number of impressions served. The oldest and
easiest measure to capture remains one of the
most commonly used techniques to assess
campaign performance (Chief marketer 2011
Interactive marketing Survey, 2011; Digital
Display Advertising 2010, Collective/ Advertiser
Perceptions, 2010), despite ample evidence of
its irrelevance as a meaningful measure (Natural
Born Clickers, comScore with Starcom uSA and
Tacoda, 2009/2010; CTR: Brand marketing’s most
misleading measure,
Collective 2010).
The case against CTR is overwhelming. The
comScore/Starcom study showed that only 16%
of all Internet users in 2010 clicked on a display
ad in a month, down from 32% a year earlier.
Collective’s own examination of one billion
advertising impressions served in the first months
of 2011 revealed that 99% of stable user cookies
examined never clicked on an ad, and that those
who did were more than two times as likely to
click again in the future. The study also estimated
that as many as 20% of the clicks were accidental,
while Click Forensics estimated in October of
2010 as many as 23% of clicks were fraudulent.
Further, a study conducted by Collective of 100
campaigns showed no correlation between CTR
and brand lift nor purchase intent as measured
by independent post-impression surveys. Hence,
optimization of campaigns to achieve higher CTR
may in fact be reducing brand ROI.
CTR is also an easy metric for third parties
to manipulate by running ads in high-click
environments, such as gaming and mobile sites,
where users will be more likely to accidentally
click on the advertisement. These accidental clicks
often frustrate the user by taking them away from
the content they were viewing and placing them
on a landing page for the brand. In these cases,
the frustrating experience may create negative
associations with the brand, causing users to be
less likely to convert than if no advertising had
been done at all.
clicK tHrougH rAte //
causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness
6
wHAt you getRIgHT AuDIENCE
RIgHT mEDIA
RIgHT CREATIvE
clicKers don’t Buy.
definitionClicks per impression.
ProsAlways on.
No conversions required.
Proxy for engagement, as the ad
must be visible to be clicked.
BottoM lineDigital advertising’s oldest
metric is also its most misleading.
consvery few people click.
Clickers rarely buy.
most clicks are accidental
or fraudulent.
Easily gamed by placing ads near
high click content (e.g., games).
Last click is a derivation of the Click through rate
born of “lower funnel” search advertising, which
assigns credit only to the last click that occurred
within a designated period of time prior to the
desired outcome (the look-back window). This
method disregards all clicks where no conversion
event followed, thus removing all accidental clicks.
However, by assigning 100% of the credit to the
last click this method fails to account for all the
previous advertising messages the user consumed
prior to making the last click, and thus tends to
undervalue display and video advertising. Further,
many users click on search advertisements as a
means of navigating to the product home page,
and would have clicked on an organic link had the
search advertising not appeared. In this fashion,
last click attribution often significantly over-values
search advertising.
The 2008 Atlas study referenced earlier cast
further doubt on Last click methodologies, point-
ing out that, “between 93% and 95% of audience
engagements with online advertising receive no
credit at all when advertisers review the ROI on
their campaigns,” because of misplaced emphasis
on Last click attribution. Quotes the study, “The
“last ad” model forces marketers to place greater
importance on the aspects of their advertising that
support the model, rather than the aspects that
support their advertising success.”
tHe study went on to drAw tHe following MetAPHor:
To illustrate the faulty logic of the model,
imagine that you’re standing in the grocery store
knowing precisely what you want to buy. You’ve
seen the product ads on TV, the full-page ad in
a magazine, and a full color mailer that actually
made you hungry just looking at the pictures.
You’ve even clipped a coupon and brought it
with you to the supermarket. When you ask the
grocery clerk where to find the specific item, he
smiles, points and says, “Aisle five.” Off you go
to aisle five, find the item, pay, and leave the
store. If you applied the “last ad” model to this
scenario, the grocery clerk would get 100% of
the credit for your purchase (no wonder he’s
smiling). As a result, marketers would invest
heavily in grocery clerks, and they’d pull their
advertising dollars from the marketing channels
that actually piqued your interest or moved you
through the funnel toward the purchase.
lAst clicKAttriBution //
causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness
7
wHAt you getRIgHT AuDIENCE
RIgHT mEDIA
RIgHT CREATIvE
disPlAy is often undervAlued.
definitionLast ad clicked prior to outcome.
Prosmeasures only productive clicks.
Difficult for intermediaries to game.
Can be linked to ROI.
BottoM lineuseful for ‘infomercial’ products.
consAds that build awareness or intent
are given no credit.
Some users would have
converted anyway.
Sometimes called the view-through conversion
rate (vTR), last impression attribution assigns
100% of the credit to the last advertising
impression served (or viewed) within a given
period of time prior to the desired outcome. In
some cases, conversions where the user clicked
are not attributed to viewed impressions, in
other cases they are. In contrast to the last click
attribution method, the look-back window is often
much longer (7, 14, 30 or even 60 or 90 days). No
clear standard exists within the industry. google,
for example, defines VTR in its Help Center as “a
measure of the number of online conversions that
happened within 30 days after a user saw, but did
not click, a display ad.”
Last impression attribution was introduced to
overcome shortcomings in last click attribution by
ensuring that viewed impressions also received
credit for conversions. In practice, however, it
gives far too much credit to digital advertising.
When the standard is an impression delivered,
the user may not even have been able to see
the advertisement that receives credit. When
the standard is a ‘viewed’ impression then while
the advertisement may be visible, there is no
guarantee that the consumer was influenced
by the advertisement to complete the desired
outcome. This leads to retargeting receiving far
more credit under last impression attribution
schemes than it should, due to the fact that
the users would have completed the desired
outcome regardless of whether or not they saw an
advertisement, yet the advertisement gets 100%
of the credit anyway.
Last impression attribution is also very easy to
manipulate (even more so than click through rate).
Intermediaries are incentivized to target users
who are likely to convert, regardless of whether
they will be influenced by the advertisement, with
the cheapest media buy possible. In the worst
case scenario, when an advertiser works with
only one intermediary, that provider can simply
hit every stable cookie with a single poor quality
advertisement once per N days (where N is the
look-back window) and they will receive credit for
close to 100% of conversions.
lAst iMPression //
1 Determining if an advertisement is visible remains a challenge for many vendors. The vendor must run code in the browser to determine if, and for how long,
an advertisement is viewable to the user. Such code is often confounded by iFrames, a publisher technology that quarantines advertisements.
See www.adexchanger.com/data-driven-thinking/viewable-impression for more details.
causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness
8
definitionLast ad viewed prior to outcome.
Prosgives credit to views not clicked on.
BottoM lineEncourages advertising to consumers
who would have converted anyway.
consAds do not have to influence
consumer to receive 100% credit.
Quality of media, placement and
creative have little impact.
Easily gamed through ‘spray & pray’
media buying.
Overvalues retargeting (would have
bought anyway).
wHAt you getRIgHT AuDIENCE
RIgHT mEDIA
RIgHT CREATIvE
Research from Collective’s Data Sciences team
explores the viability of a Causal Attribution
measurement scheme that, through a carefully
constructed experiment, directly observes
the change in desired outcomes caused by
incremental digital advertising spend. Because
this method uses an A/B test experimental
design established prior to the incremental
advertising, the results are unbiased by organic
conversions and by other advertising spend,
including spend offline.
Further, because the design is based on random
audience groups, rather than randomly selected
impressions, Causal Attribution can measure
the cumulative effect of multiple advertising
impressions over a period of time on individual
users. This is accomplished without having
to waste any impressions on public service
announcements (PSAs), a common requirement
of other experiment based digital measurement
approaches. When combined with a measure
of desired outcome value (e.g., profit from a
conversion), Causal Attribution can measure
the true return on investment (ROI) driven by
incremental advertising spend.
In the sections that follow, we will:
// Illustrate how Causal Attribution works through
a simple six step process.
// Outline the types of outcomes that can be
measured using Causal Attribution.
// Examine in detail the results of two real
campaigns where Causal Attribution was used to
quantify ROI and identify optimal audiences.
// Introduce an alternate methodology which
uses PSA advertisements.
// Outline additional applications that can
optimize creative, media and frequency.
We will then conclude with a more thorough
discussion of the benefits (and limitations) of
Causal Attribution for advertisers, and a discussion
of how widespread adoption of this best practice
could impact the industry as a whole.
cAusAl AttriBution //
causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness
9
siMPle, scientific, unBiAsed, true roi.
definitionExperiment measures outcomes
caused by advertising.
ProsDirectly quantifies ROI.
unbiased by all other advertising.
Transparent and easy to administer.
Provides rich audience analytics.
BottoM linemeasures causality, not correlation,
providing a true measure of ROI.
consRequires large audience database.
Does not always produce results
that are statistically significant.
wHAt you getRIgHT AuDIENCE
RIgHT mEDIA
RIgHT CREATIvE
Step 1 // define tHe Audience cloud
We first limit the experiment to cookies that
are likely to persist for the duration of the
experiment. This is done to limit the impact of
cookie deletion. At Collective, we define a stable
cookie to be one seen at least once within the
last 28 days and on at least two separate days
over the life of the cookie. This ensures that the
cookie was recently seen, and was persistent for
at least a period of two days (and so unlikely to
have been automatically deleted by the browser
after the session).
We have found that cookies seen on at least two
separate days of the prior 28 are 22 times more
likely to appear the following day than cookies
seen on only one day. On average at Collective,
this provides a stable universe of approximately
200 million users, which we refer to as our
Audience Cloud.
Step 2 // divide tHe users into test And control grouPs
Before the campaign begins, we randomly divide
the entire Audience Cloud into test and control
groups using a robust random number generator.
This split is repeated with a different initial seed
for every advertising campaign, ensuring that
the randomization is unique for each advertiser
and there is no cross-experiment pollution. In
practice, we typically choose anywhere from 5%
to 50% of the cookies to be in the control group,
depending upon the length of the campaign and
the frequency of the desired outcome absent
advertising.
given an Audience Cloud of 200m users, this
means our control group will consist of anywhere
from 10m to 100m users. These sample sizes
ensure that the test and control groups will be
balanced across all other influencing factors, such
as audience demographics and exposure to other
advertising (either digital or traditional).
Step 3 // deliver AdvertiseMents only to tHe test grouP
We then begin the advertising campaign, and
ensure that cookies in the control group are
never exposed to the advertising. In practice,
this is done by negatively targeting an audience
segment in our ad-serving engine that includes all
of the cookies in the control group. Whenever an
impression arrives for a user in the control group,
the campaign is excluded from the potential set of
campaigns to be shown.
Note that for the purposes of measuring the
impact of the campaign we do not need to
adjust the experiment to control for any audience
targeting, media selection, creative optimization
or frequency capping was used in the campaign.
This is possible because we are comparing the
entire control group with the entire test group,
regardless of who the ad was delivered to.
Being able to ignore these other factors drastically
simplifies the process of analyzing the experiment,
and also ensures that we can run the experiment
on any campaign without disruption, allowing for
ongoing monitoring and optimization.
How cAusAlAttriBution worKs //The great French physicist and mathematician, Henri Poincaré, once said that “experiment is the sole source of truth; it alone can teach us something new; it
alone can give us certainty.” Without experimenting, no amount of conjecture or analysis of data can prove a scientific theory. Similarly, without experimenting, we
cannot prove that advertising has caused a desired outcome, nor can we truly optimize our ROI.
Causal Attribution works by creating an experiment where the only difference between two otherwise identical pools of users is that one pool (the test) was
potentially exposed to advertising, and the other pool (the control) was not. This design ensures that it was our decision to advertise to the test group that caused
any statistically significant difference measured in the desired outcome rates of the test and control pools.
In practice, there are some subtle and important decisions to be made in conducting a Causal Attribution experiment. The following six steps outline the process
and illustrate how Collective conducts these experiments on behalf of advertisers.
causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness
10
2 Negatively targeting the control group also allows the campaign to deliver outside of the universe of stable cookies, which is often desirable for meeting
delivery goals and maximizing reach. One complication of this approach is that when computing ROI we must remove spend on impressions outside of the test
and control groups (see step 6).
Step 4 // oBserve desired outcoMes on All users
During the course of the campaign, and for a
period of time after the campaign concludes we
observe all desired outcomes on all cookies in
the Audience Cloud.
It is critically important that we observe all
desired outcomes, as the outcome rate in
the control group serves as our baseline for
evaluating lift caused by the advertisement.
For example, applying a last view or last click
attribution scheme would discard those desired
outcomes that did not have a view or click event
within a selected look-back window. Thus,
the control group conversion rate would be
artificially reduced to 0% and the experiment
would be invalidated.
Similarly, all desired outcomes should be counted
in the experiment, regardless of the path that
the user took to complete the desired outcome.
For example, if we are measuring online product
purchases, then we should track all purchases
online regardless of how the user clicked to arrive
at the checkout page – be it directly through the
uRL, through an organic or sponsored search, or
by clicking on a display or video advertisement.
This ensures that we capture the full impact of the
advertising impressions by including all purchases
that the advertising could have influenced.
Step 5 // MeAsure tHe desired outcoMes rAtes
After the campaign has concluded and the
follow-up period has elapsed we can compute
the rate of desired outcomes, defined as the
total number of outcomes divided by the total
number of cookies, in both the test and the
control groups. Note that, depending upon the
type of outcome being measured, you can either
allow cookies to complete the desired outcome
more than once or only once. For example, if
the outcome is a purchase, then you should
count all purchases from each cookie. If instead,
the outcome is a survey response or a site
registration, you may wish to count the outcome
only once per cookie.
Step 6 //coMPute tHe cAusAl lift
We define the causal lift of the campaign as the
lift observed in the desired outcome rates in the
test group over the control group. For example,
suppose we find that 0.12% of users in the test
group purchased the product, but only 0.10%
of users in the control group purchased the
product. given that these two user pools were
randomly chosen from the same population, and
the only difference was our decision to advertise
to the control group, we can conclude that the
advertising caused users to be 20% more likely to
convert over the time period in question.
sPlit test & control
(Audience cloud 200 Million cooKies)
cAMPAign runs MeAsure outcoMes
ADVERTISE TO TESTOBSERVE OUTCOMES
TEST (REMAINING 90%)
CONTROL (RANDOM 10%)
TEST OUTCOME RATE
# TEST OUTCOMES# TEST COOKIES
# CONTROL OUTCOMES # CONTROL COOKIES
CONTROL OUTCOME RATE
=
=
ADVERTISE TO TESTOBSERVE OUTCOMES
TEST (REMAINING 90%)
CONTROL (RANDOM 10%)
TEST OUTCOME RATE
# TEST OUTCOMES# TEST COOKIES
# CONTROL OUTCOMES # CONTROL COOKIES
CONTROL OUTCOME RATE
=
=
ADVERTISE TO TESTOBSERVE OUTCOMES
TEST (REMAINING 90%)
CONTROL (RANDOM 10%)
TEST OUTCOME RATE
# TEST OUTCOMES# TEST COOKIES
# CONTROL OUTCOMES # CONTROL COOKIES
CONTROL OUTCOME RATE
=
=
online conversions orActions (Proxies):
Consumer online purchasing or other desired online
activities, such as visiting a website, completing a
registration, viewing a video, etc., can be measured
using outcome pixels, such as DART Spotlight
activities or AmP audience pixels. In general, these
pixels should be closely linked to potential revenue
generation, and should be widely accessible to
consumers, regardless of their path to conversion.
This ensures the full impact of the advertising spend
is captured, and also helps to ensure statistical
significance of results.
off-line conversions:
Though a less immediate measure, off-line sales can
be applied to Causal Attribution logic through the
import of anonymized point of sale and/or credit
card transaction data, available through third party
vendors. Similarly, an advertiser’s internal CRm
purchase log data can be anonymized and moved
online to a cookie through a data management
platform. This approach allows the advertiser to
quantify the impact their online advertising is having
on offline sales without having to use any personally
identifiable information (thus protecting consumer
privacy) in the design or analysis of the experiment.
BrAnd MeAsureMent:
An important and often overlooked metric in
digital advertising today is evaluating the impact
of advertising spend online on brand awareness,
message recall and future purchase intent.
Custom surveys executed through rich media ad
units can record the responses directly to a user
cookie so that the (self-reported) results of a brand
measurement survey can be observed across
the test and control groups . At the conclusion
of the experiment, any lift observed in desired
survey response rates in the test group over the
control group can be credited to the incremental
advertising spend.
desired outcoMes tHAt cAusAl AttriBution cAn MeAsure //cAusAl AttriBution cAn MeAsure A cross section of Advertising oBjectives:
3 Note that for brand measurement, it can be significantly more cost effective to conduct the Causal Attribution study using a creative based approach with
public service announcements (PSAs). This limits the audience where surveys should be collected to just those users who received an advertisement in the test
group or a PSA in the control group. See the PSA methodology section for more information.
causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness
12
The Collective Data Sciences team analyzed the
behaviors of 200 million stable audience cookies
across 14 live advertising campaigns during a
period from October 2011 through January
2012. In all, 380 million advertising impressions
were examined. The audience profiles examined
contained an average of 70 data attributes, such as
location, demographics and enthusiast behaviors.
Two Selected examples follow.
exAMPles of cAusAl AttriBution tests on live cAMPAigns //
In this test, advertisements for a national hotel
chain were analyzed for their ability to drive online
bookings. The test analyzed a test group of 180
million cookies and a control group of 20 million
cookies. At the end of two months, the exposed
group converted at a 0.0050% rate, and the
control group converted at a 0.0044% rate. This
means that the advertising spend caused users
to be 14% more likely to convert than they would
have otherwise. using a difference in propor-
tions test , we find that the p-value for this lift is
0.00005, indicating that it is highly statistically
significant, with less than a 1 in 20,000 chance of
observing a lift this high randomly.
What is more, we can convert this lift directly
into a return on investment (ROI). The test group
converted at 0.006 percentage points higher than
the control group. multiplying this rate by the
number of cookies in the test group we can con-
clude that there were 1,099 additional conversions
caused by this campaign. The client valued online
conversions at $30 each, therefore, this additional
advertising spend generated $32,970 worth of
incremental value for the hotel chain.
The campaign cost $25,000, and so the ROI for
the advertiser was 32%. meaning that, for every
$100 the advertiser invested in this campaign, they
made $32 of incremental profit.
The experiment becomes even more interesting
as the nature of the converting customers is exam-
ined and we compare Causal Attribution results
for audience segments to performance measured
using other attribution methods.
For example, we can segment the Audience Cloud
by the context of the pages each user most often
frequents. We call these segments enthusiast
behaviors, as they indicate that a cookie is enthu-
siastic about a category of content (e.g., music,
sports, food). Because the test and control groups
were randomly selected from the entire Audience
Cloud, each of these enthusiast groups will be ran-
domly split between the test and control popula-
tions, and we can analyze the causal lift generated
within each segment.
In the chart to the right, we show how each enthu-
siast group compares to the average by plotting
the percentage lift (on the horizontal axis) for a
particular metric for each group.
The metrics shown are the control conversion rate
in gray (how often the control group converts ab-
sent advertising), the click through rate in pink and
the causal lift in blue (how much more likely users
are to convert when exposed to advertisements).
The enthusiast behaviors where we observed at
least 500,000 users in the control groups have
been sorted on the vertical axis by the control
conversion rate lift in descending order.
We find that the users most likely to book hotel
rooms in the control group are those reading
about real estate and, not surprisingly, travel.
These groups are also among the highest in click
through rate. Yet their causal lift is actually nega-
tive, indicating that advertising to these groups
will not make them any more likely to convert than
they already were.
cAse study trAvel BrAnd //
causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness
14
teSt ReSULtS: tRAVeL BRAND //While a 32% ROi is a pOWeRful Result (eveRy $100 spent dROve $32 Of incRemental pROfit), audience taRgeting cOuld have have geneRated an ROi as high as 150%.
14%
0.00005
1,099
$30
$32,970
$25,000
32%
criteriA
# cooKies
# conversions
conversion rAte
conversion lift
stAtisticAl significAnce
conversions cAused
vAlue of A conversion
vAlue generAted
sPend
roi
CONVERSIONS / COOKIES
(TEST RATE - CONTROL RATE ) / CONTROL RATE
P - VALUE FOR CONVERSION LIFE EXCEEDING 0 %
(TEST RATE - CONTROL RATE) X TEST COOKIES
CONVERSIONS CAUSED X VALUE OF CONVERSION
(VALUE GENERATED - SPEND) / SPEND
control
20
882
0.0044%
test
180
9,037
0.0050%
causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness
15
MEASUREMENT
MEASURE LIFT (%)
CLICKERS
CONTROL CONVERTERS
CASUAL LIFT
GAMES
FASHION
FINANCE
MUSIC
TECHNOL-
HEALTH
SCIENTIFIC
AUTOMO-
POLITICAL
FAMILY
SHOPPING
FOOD
WEATHER
SPORTS
TRAVEL
REAL
-100 -50 0 50 100 150
While real estate and travel enthusiasts book many hotel rooms, they Were not influenced by the advertising. PeoPle reading about fashion and shoPPing Were.
clicKers
control converters
cAusAl lift
-100 -50 0 50 100 150
MeAsure lift (%)
reAl estAte
trAvel
sPorts
weAtHer
food
sHoPPing
fAMily
PoliticAl
AutoMotive
scientific
HeAltH
tecHnology
Music
finAnce
fAsHion
gAMes
In contrast, near the bottom of the chart we find
that in the control group those users reading
about fashion are less likely to book hotel rooms
than those reading about travel, and are less likely
to click on the hotel chains ads. However, they
are more than 50% more likely to convert when
they were exposed to the campaign. Thus, we
can conclude that the advertising is significantly
influencing these users despite their low response
rates using traditional attribution methods.
What then is the optimal audience profile for this
hotel chain? To answer that question, we can
analyze the return on investment for every
audience segment.
While the brand achieved a 37% ROI on the
campaign as a whole, there are large audience
groups where the ROI is significantly higher. Users
who are older and users with higher incomes
generate ROIs ranging from 50% to 125%.
Similarly, there are large pockets of users by
geography, browser and enthusiast behavior who
generate much higher ROIs.
By selectively targeting those segments exhibiting
higher ROIs, an advertiser can drive materially
higher performance. Further, sophisticated look
alike modeling techniques can be employed to
score every user in the audience cloud from lowest
expected ROI to highest expected ROI, using
a multitude of factors including demographics,
geography and behavior online. Then, a custom
audience group can be created of just those users
who we expect to deliver the highest ROI for the
advertiser.
cAse study : trAvel BrAnd (continued)
causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness
16
18-2
425
-34
35-4
445
-54
55-6
464
+
00-2
525
-50
50-7
575
-100
100-
150
150-
250
250+
FEM
ALE
MAL
E
ES C
ENTR
ALW
S CE
NTRA
L
EN C
ENTR
ALS
ATLA
NTIC
MID
DLE
ATLA
NTIC
PACI
FIC
MOU
NTAI
NW
N CE
NTRA
LNE
W E
NGLA
ND
CHRO
ME
IEFI
REFO
X
FITN
ESS
TRAV
ELRE
LIGI
ONRE
AL E
STAT
E
FAM
ILYSP
ORTS
AUTO
MOT
IVE
FOOD
WEA
THER
GAM
ESM
USIC
POLI
TICA
LSC
IENT
IFIC
FINA
NCE
HEA
LTH
HOU
SEH
OLD
TECH
NOLO
GYPE
TSSH
OPPI
NGFA
SHIO
NGA
DGET
SJO
BS
AUDIENCE SEGMENT
150
100
50
0
-50
-100
-150
To illustrate the impact on performance that op-
timizing to ROI can have, suppose the advertiser
selected the top 20% of audience segments by
their conversion rate alone. The advertiser would
achieve a conversion rate of 0.093% and a causal
ROI of 33%. If instead they were to select the top
20% of audience segments by ROI, their conver-
sion rate would drop to 0.059% (a 35% decrease),
significantly hurting their last view impression
conversion rate. However, their ROI would rise to
104% (a 216% increase), generating more than 3
times as much value per dollar invested.
This is illustrated in the chart to the right, in which
each audience segment is a circle arrayed on a
plot where the x-axis is that audience’s conversion
rate in the control group, and the y-axis is that
audience’s ROI. Last view conversion attribution
analysis includes the 6 pink audience groups in
the top 20%, whereas an ROI analysis correctly
identifies the 6 blue audience groups instead. (The
4 split color circles in the upper right were selected
by both strategies.)
This finding also holds true for click through rate.
Were the advertiser to select the top 20% of audi-
ence segments by their CTR alone, they would
achieve a CTR of 0.048% and a causal ROI of 45%.
If instead they were to select the top 20% of audi-
ence segments by ROI, their CTR would drop to
0.026% (a 46% decrease), yet their ROI would rise
to 104% (a 129% increase).
cAse study : trAvel BrAnd (continued)
causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness
17
KeyoPtiMize to conversion rAteoPtiMize to roioPtiMize to BotH
0.002 0.004 0.006 0.008 0.010 0.012
-50
0
50
100
CONVERSION RATE (%)
RETU
RN O
N IN
VES
TMEN
T (%
)
oPtimizing on causal attributiondrives 216% higher rois than last vieW.
In this test, advertisements for a fashion retail
brand were analyzed for their ability to drive
registrations for an online website. The test
analyzed 180 million exposed cookies and a
control group of 20 million. The exposed group
converted at a 0.0185% rate, and the control
group at a 0.0184% rate, meaning that the
test group that included users exposed to the
advertising was just slightly more likely to register
for the website as those who had not been
exposed to the ads. The p-value for this lift is 0.37,
indicating that it is not statistically significant, with
more than a 1 in 3 chance of observing lift at this
level by chance alone.
The campaign caused 188 conversions, valued at
$10 each, for a total of $1,880 of value generated
for the advertiser. The campaign cost $15,000,
demonstrating a negative 87% return
on investment.
This experiment exposed for the client that
their current advertising campaign was not
driving enough registrations to justify their
media spend. It is critical to observe that in this
case the standard attribution models painted
a very different picture. Whereas the standard
performance measures being used by the client
led them to believe that this campaign was one
of their best performing, our Causal Attribution
study revealed that it was actually having almost
no impact on online registrations. This ability to
definitively identify campaigns and optimization
strategies that are not working is one of the
greatest advantages of using Causal Attribution.
For example, the control converters, those who
converted without being advertised to, were
young and generally mid-to-high income, whereas
the ‘clickers’ were older and lower income. In
contrast, the blue causal lift lines better represent
the actual responsiveness of a group to online
ads. In this case, causal lift was fairly consistent
regardless of age and household income.
This audience analysis highlights how misleading
existing attribution solutions can be. By targeting
high-income young audiences, a partner could
drive a very high last impression conversion rate.
By targeting middle-income older audiences, a
partner could drive a very high click through rate.
But neither strategy would materially impact causal
lift – that is, convince consumers to register online
who would not have registered anyway.
In general, we find that audience groups that
are naturally predisposed to be likely to convert
or click are rarely the groups that will be most
influenced to convert by online advertising. Only
through the analysis of Causal Attribution results
can an advertiser truly discover who the optimal
audience is for a given advertising campaign.
The results of this analysis led the advertiser to
conclude that they should experiment with new
creative strategies, and use Causal ROI to test
which audience groups are most receptive to
each creative.
cAse study retAil BrAnd //
causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness
18
teSt ReSULtS: RetAiL BRAND // tRaditiOnal metRics (last click, last vieW) pOsitiOned this campaign as One Of the tOp 2 On this adveRtiseR’s buy, yet causal attRibutiOn pROved that it Wasn’t deliveRing value. the adveRtiseR is using the Rich RepORting fROm the causal attRibutiOn analysis tO Revisit theiR cReative and audience taRgeting stRategies.
1%
0.37213
188
$10
$1880
$15,000
-87%
criteriA
# cooKies
# conversions
conversion rAte
conversion lift
stAtisticAl significAnce
conversions cAused
vAlue of A conversion
vAlue generAted
sPend
roi
CONVERSIONS / COOKIES
(TEST RATE - CONTROL RATE ) / CONTROL RATE
P - VALUE FOR CONVERSION LIFE EXCEEDING 0 %
(TEST RATE - CONTROL RATE) X TEST COOKIES
CONVERSIONS CAUSED X VALUE OF CONVERSION
(VALUE GENERATED - SPEND) / SPEND
control
20
3,689
0.0044%
test
180
33,389
0.0185%
causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness
19
MeAsureMent
clicKers
control converters
cAusAl lift
AGE HHI Measurement
Clickers
Control Converters
Causal Lift
150
100
50
0
-50
-100
18-24 25-34 35-44 45-54 55-64 0-25 25-50 50-75 75-100 100-150 150-250 250+65+
inverses : converters Were young but clickers Were older.
converters Were Wealthier;clickers, less so.
An alternative to the methodology outlined in
this paper is to execute a Causal Attribution
experiment through the use of public service
announcements (PSAs) in the creative serving
engine. Steps 1 & 2 are the same, but in step 3
instead of negatively targeting the control group,
we serve a public service announcement (PSA)
to the control group, and serve the campaign
advertisement to the test group. Note that the
test and control groups must still be cookie based,
rather than impression based, or else cookies
will move back and forth between the test and
control groups polluting any cookie based
conversion analyses.
The main advantage of serving a PSA is that it
narrows the scope of the experiment from all
200m cookies in the Audience Cloud to just those
served an advertisement (either PSA or from the
campaign) in the experiment. In some cases, this
can improve the statistical significance of the
results. It can also dramatically reduce the number
of desired outcomes that must be gathered
when measuring brand lift impact. However, the
requirement to serve a PSA is cost prohibitive,
especially when conducting Causal Attribution on
a routine basis with a large control group.
An additional advantage to using a PSA is that
control over the audience targeting and ad
decisioning is no longer needed. Instead, the
experiment is conducted purely in the creative
serving, ensuring that the control group only
receives PSA advertisements. This allows Causal
Attribution to be extended across media buys
spanning multiple partners and channels.
One common misconception is that in conducting
a creative-based Causal Attribution experiment
one can analyze the impact that frequency has on
the desired outcome rate. This can be attempted
by evaluating the lift in test over control
conversions by cookie stratified by the frequency
with which the cookie was reached (by either
the PSA or real advertisement). Thus, one might
observe that the lift in conversions for users with
a frequency of 5 was 20% higher than the lift in
conversions for users with a frequency of 1.
This type of analysis, however, is seriously flawed.
The primary driving factor behind this measured
frequency is not the decision to serve the
advertisement, but is instead the user’s frequency
of use of the Internet, and the percentage of
those impressions purchased. Thus, any difference
in conversion rates by frequency could be caused
by these underlying biases rather than by having
been exposed to the advertisement
additional times //
PsAMetHodology//
causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness
20
creAtive AnAlysis
The choice and execution of creative, be it display
or video, is undoubtedly one of the most influen-
tial decisions an advertiser can make in executing
a digital advertising campaign. Rather than relying
upon intuition alone in designing creative, Causal
Attribution allows an advertiser to test multiple,
potentially radically different, creatives to identify
which audiences are most influenced by each.
Thus, an advertiser can increase the effective reach
of their campaign and truly serve the right audi-
ence with the right creative. This is accomplished
by dividing the test audience into random groups,
each of which is only ever exposed to one type of
creative. Combined with dynamic creative optimi-
zation, a given creative can be further optimized to
achieve maximal ROI.
MediA AnAlysis
The choice of where an advertisement is run is
also a critical decision in any digital campaign.
Causal Attribution experiments can be designed
to test the impact that advertising in a premium
environment has over advertising elsewhere on the
Internet, in either longer tail inventory or with user
created content. Again, this is accomplished by
dividing the test group into multiple pools, each of
which is targeted only in certain ad environments.
Similarly, one can compare inventory purchased
directly from publishers with inventory purchased
indirectly through exchanges.
frequency AnAlysis
Causal Attribution can also be used to quantify
the effect of frequency (how many times each
cookie is shown an advertisement) on ROI. This
is accomplished by dividing the Audience Cloud
into several smaller pools (e.g., 2 million cookies
each). One is kept as a control group, and the oth-
ers are targeted in the same manner with varying
frequency caps. For example, we might create four
small pools:
// A control pool that will never be shown
an advertisement.
// A pool with a 1 per 1 day frequency cap.
// A pool with no frequency cap.
Keeping each pool relatively small increases our
chances of hitting cookies in the pools at close to
the desired frequency cap rate.
furtHerAPPlicAtions //In addition to analyzing ROI at the campaign and audience profile level, there are three significant ad-
ditional applications of Causal Attribution. While we will only briefly cover them in this white paper, they
will be the basis of additional research in the future. Note that in the following sub-sections we describe
each experiment in isolation, but they can in fact be conducted simultaneously using multivariate
testing strategies.
causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness
21
4 Statistical significance is governed by three factors: i) the size of the control group, ii) the amount of lift generated by the campaign, iii) the proportion of users
completing the desired outcome absent advertising. In practice, we have found that the third factor is most often to blame for inconclusive results, and caution
against running causal attribution studies based on very rare desired outcome events.
tHe Benefits And liMitAtionsfor cAusAl AttriBution for Advertisers //
tHe Benefits
The present study establishes that Causal Attribu-
tion provides a compelling system for evaluating
digital advertising ROI that avoids many of the
pitfalls present in other attribution solutions. The
following are some of the principle benefits of
Causal Attribution to advertisers:
// Directly quantifies return on investment (value
generated from spend)
// unbiased by all other advertising (occurs in
both test and control groups)
// Transparent and easy to administer (no propri-
etary algorithms)
// Provides rich audience analytics (identify con-
sumers who will be influenced)
// Effective across all channels where cookies are
used (display, video, mobile web)
tHe liMitAtions
Simply put, no other attribution system can pro-
vide all of these advantages. However, there are
some limitations of Causal Attribution that should
be noted:
// Depends on browser cookies, which are often
deleted (biases ROI low)
// Limits the reach of a given campaign by the size
of the control group
// Does not always produce results that are statis-
tically significant
tHe conclusion: A ProPosAl for A new industry MeAsureMent stAndArd
At Collective, we firmly believe that the benefits to
advertisers of Causal Attribution far outweigh the
limitations. Combined with our findings on how
current methodologies can produce false success
signals that misguide marketers into making poor
media planning and optimization decisions, we
believe that Causal Attribution should be adopted
as an industry standard.
causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness
22
The digital advertising industry has become
enamored with data, technology and algorithms.
Billions of data points are analyzed, inventory is
dynamically selected through real time bidding,
and sophisticated audience, media and creative
optimization strategies are deployed. And yet, all
of this investment is frequently evaluated using
misleading attribution methodologies. When
coupled with highly competitive and sophisticated
media partners using questionable optimization
tactics, these metrics are severely hindering
the industry.
To quantify the economic impact of the problem
today, consider that Forrester estimates in their
“uS Interactive marketing Forecast, 2011 to 2016”
report that the market for display advertising was
close to $12 billion in the uS in 2011. Assume
that 75% of this spend was for direct response
campaigns. Further, conservatively assume that
20% of that spend was wasted due to misleading
measurement systems. Then this implies that $1.8
billion of spend was wasted in 2011 in the
uS alone.
Widespread adoption of Causal Attribution would
have an enormous impact on the entire advertis-
ing ecosystem. Digital advertisers would rapidly
cut the fraction of their spend that is not effective,
and be willing to pay significantly more for the
right audiences and ad environments that deliver
high ROIs. CmOs would be willing to spend more
of their advertising budgets in digital where they
can concretely measure the value the advertis-
ing is generating. Publishers would respond by
limiting the supply of lower quality and less visible
advertising inventory, and instead focus on provid-
ing advertisers with impressions that have a higher
probability of influencing consumers.
Ultimately, more advertising dollars would flow
into digital channels, Advertisers would get higher
returns on their investment and publishers would
be rewarded for generating higher quality engag-
ing content. Finally, consumers would win by en-
joying higher quality content with fewer pay-walls,
and by receiving more relevant advertisements
that could truly help their purchase decisions.
How cAusAl AttriBution could cHAnge tHe industry //
causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness
23
causal attribution measurement will allow advertisers, for the first time, to accurately evaluate how their online advertising affects the behavior of specific audiences and measure the roi generated from their advertising spend.
“”
– Jeremy stAnley, sVp product And dAtA sciences, collectiVe
MeAsureMentstrAtegies //
definitionLast ad clicked prior to outcome.
Prosmeasures only productive clicks.
Difficult for intermediaries to game.
Can be linked to ROI.
BottoM lineuseful for ‘infomercial’ products.
consAds that build awareness or intent
are given no credit.
Some users would have converted anyway.
lAst clicKAttriBution //
wHAt you getRIgHT AuDIENCE
RIgHT mEDIA
RIgHT CREATIvE
definitionClicks per impression.
ProsAlways on.
No conversions required.
Proxy for engagement, as the ad
must be visible to be clicked.
BottoM lineDigital advertising’s oldest metric is
also its most misleading.
consvery few people click.
Clickers rarely buy.
most clicks are accidental or fraudulent.
Easily gamed by placing ads near high
click content (e.g., games).
clicK tHrougH rAte //
wHAt you getRIgHT AuDIENCE
RIgHT mEDIA
RIgHT CREATIvE
definitionLast ad viewed prior to outcome.
Prosgives credit to views not clicked on.
BottoM lineEncourages advertising to consumers
who would have converted anyway.
consAds do not have to influence consumer
to receive 100% credit.
Quality of media, placement and creative
have little impact.
Easily gamed through ‘spray & pray’
media buying.
Overvalues retargeting (would have
bought anyway).
lAst iMPressionAttriBution //
wHAt you getRIgHT AuDIENCE
RIgHT mEDIA
RIgHT CREATIvE
causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness
24
definitionComputer algorithm assigns credit
for outcomes by analyzing data.
ProsCan overcome some limitations
in simpler metrics.
BottoM line“Pay no attention to the man behind
the curtain.”
consOften confuses correlation for causation.
Requires measuring all user interactions
in all channels.
No industry standard likely to be developed.
Only programmers really know how
they work.
wHAt you getRIgHT AuDIENCE
RIgHT mEDIA
RIgHT CREATIvE
? ? ? ? ?? ? ? ? ?? ? ? ? ?
AlgoritHMicAttriBution //
definitionuser determines a weighting scheme
for mixing CTR, last click and last view.
ProsAverages out some of the misleading
facets of simpler metrics.
gives the advertiser or agency “knobs
and dials“to control.
BottoM linegarbage in, garbage out.
consPoor metrics cannot be mixed together
into a good metric.
Choice of weights, even when informed by
data, is highly subjective.
Still provides no direct link to ROI.
weigHtedAttriBution //
wHAt you getRIgHT AuDIENCE
RIgHT mEDIA
RIgHT CREATIvE
definitionExperiment measures outcomes
caused by advertising.
ProsDirectly quantifies ROI.
unbiased by all other advertising.
Transparent and easy to administer.
Provides rich audience analytics.
BottoM linemeasures causality, not correlation,
providing a true measure of ROI.
consRequires large audience database.
Does not always produce results
that are statistically significant.
wHAt you getRIgHT AuDIENCE
RIgHT mEDIA
RIgHT CREATIvE
cAusAlAttriBution //
causal attribution Positioning A Better Industry Standard Measure of Display Advertising Effectiveness
25
ABout collective //Collective intelligently connects brands to audiences with high-impact experiences across
display, video and mobile. Collective’s AmP(R) Data and media management platform
powers the ad businesses of over 50 leading media brands, including our flagship media
products, Collective Display and Collective video.(R) Collective’s complete buy-side solu-
tion, Ensemble,(Tm) provides brand advertisers with audience buying combined with rich
media and DCO.
Collective is headquartered in New York with offices in Atlanta, Boston, Chicago, Dallas,
Detroit, Los Angeles, San Francisco, London and Bangalore. Collective’s investors include
Accel Partners(R), greycroft Partners and iNovia Capital. For more information, please visit
www.collective.com.
Press contAct //Laura Colona
Director of Communications
sAles //[email protected]
99 Park Avenue, 5th Floor New York, NY 10017 888-460-9513