google analytics and adwords optimisation with gnu r

Google Analytics and AdWords optimisation withGNU R

Hinnerk Gnutzmann & Piotr Śpiewanowski

flexponsive UG

Booster Conference, 9th March 2016

About flexponsive UG

• e-commerce consulting• Big Data focus• Qualitative user testing• Academic (PhD in economics) and programming background

Contact• mailto: spiewanowski@flexponsive.net• web: https://www.flexponsive.net/• t: @flexponsive

Topic of the day

• Marketing outcomes• difficult to define• even more difficult to measure

• Before Big Data: “Half the money I spend on advertising is wasted;the trouble is I don’t know which half.” (John Wanamaker, 1838 -1922)

• With Big Data: “AdWords brand keyword ads have no measurableshort-term benefis” (Blake et al., 2015) - 100% wasted?

• Open Questions:• Incrementality Debate: Do AdWords campaings cannibalise organictraffic?

• Quality: Are bought visitors good or bad customers?• Heterogenity: Campaign effects differ between customers?

Agenda

1. Case study Brand Keyword: The Secret of vanishing AdWords ROI2. What can we do?

• attribution models• controlled experiments• GNU R & Analytics: A Dream Team

3. How to do that?• Google Core Reporting API & GNU R• GA Query Explorer• Configuring an experiment in AdWords

4. Analysis with GNU R• Data wrangling, sampling, etc.• GA replicate metrics• Regression Analysis

5. Case Study II: adClicks and rain in Bergen

Example - Skandiabanken

Example - Skandiabanken

What happened?

• the AdWord is highly relevant to the search• Navigational Query: The visitor wants to visit Skandiabanken.• Customer knows the bank and maybe even has a service in mind

• Result: Probably the best keyword in the account• Excellent CTR• Very good conversion on-site• CPC perhaps not so high

• Any questions?

• Organic result is the same!• What would you click if there was no ad?

What happened?

• the AdWord is highly relevant to the search• Navigational Query: The visitor wants to visit Skandiabanken.• Customer knows the bank and maybe even has a service in mind

• Result: Probably the best keyword in the account• Excellent CTR• Very good conversion on-site• CPC perhaps not so high

• Any questions?

• Organic result is the same!• What would you click if there was no ad?

What happened?

• the AdWord is highly relevant to the search• Navigational Query: The visitor wants to visit Skandiabanken.• Customer knows the bank and maybe even has a service in mind

• Result: Probably the best keyword in the account• Excellent CTR• Very good conversion on-site• CPC perhaps not so high

• Any questions?

• Organic result is the same!• What would you click if there was no ad?

Example - Skandiabanken (without AdWords)

ROI

Problem: SEM expenditure a function not only of the campaign, but alsoof the behavior and intent of consumer

The eBay study

• Blake et al. (2015), “Consumer Heterogeneity and Paid SearchEffectiveness: A Large Scale Field Experiment”

• Field Experiment: Does AdWords work for eBay?

• Very controversial results:1. Conventional methods used to measure the causal (incremental)

impact of SEM vastly overstate its effect.2. True effectiveness of SEM is small for a well-known company like eBay3. Click substition: When the brand keyword AdWord disappeared,

almost all the users click on the organic result4. Informative Advertising: AdWords work if a visitor gains additional

information through advertisement - AdWords had almost no effect onrevenues from existing customers - They found their own way to eBay!

What can be done? Attribution modelling

But how to know the true channel’s impact?

Attribution modelling

• a way to divide the “credit” for a sale between different marketingchannels

• if you don’t know what attribution model you are using, it’s “lastclick” => you believe the sale only depends on the last ad thecustomer saw before purchasing

• probably that’s not true: perhaps the customer had been followingthe company blog for a long time, heard friends talk off-line about theproduct, or saw many banner ads on different sides before making apurchase

• problem: no good way to decide how to “attribute” between differentmarketing channels

• results depend a lot on assumptions, which you cannot test

• similar problem: if you advertise your brick-and-mortar store on TVand on radio, what drives the customer to your store?

What can be done? Controlled experiments

• Select by random treatment and control group, for example:• Per user: A / B Testing• By Geographical Region

• Assumption: Without experiment, both groups behave similarly• Evaluation: difference in differences

• difference in the control group: Noise• difference in treatment group: Effect + Noise

• Metrics: ∆TREATED − ∆UNTREATED

• Advantages of a geographical experiment:• no multi-device tracking necessary• easy integration with external data

• Caveat: Geographical groups really need to be comparable(e.g. commuters)

Difference in Differences

GNU R and Google Analytics: Dream Team

1. Selection of the treated and control group• Install R, generate a sample with GNU R• Export: Copy & paste to AdWords

2. Data collection• Google Analytics already configured

3. Aggregation and query• In the cloud: Google Analytics Query Explorer• Integration with RGoogleAnalytics

4. Evaluation: Estimation and Visualization• All necessary functions available as packages in R

About R

• Programming language and software environment for statisticalcomputing and graphics, a dialect of S

• Quite lean; functionality is divided into modular packages• Graphics better than in most stat packages.• Useful for interactive work, but contains a powerful programming

language for developing new tools (user -> programmer)• Very active and vibrant user community; R-help and R-devel mailing

lists and Stack Overflow• Markdown packages for reproducable research and automated

reporting• It’s free!

Install R

• Open Source for Windows / Mac / Linux etc.• GNU R: https://www.r-project.org/• RStudio IDE: http://www.rstudio.com

• Cheat Sheets to help!• R Reference Card• RStudio cheatsheets

• Package management via CRAN

install.packages('RGoogleAnalytics',repos = "http://cran.no.r-project.org");

install.packages('plm',repos = "http://cran.no.r-project.org");

install.packages('ggplot2',repos = "http://cran.no.r-project.org");

Selecting Treatment Group

download.file('https://goo.gl/qVgiYp',destfile='geoid.csv');

#Kommune level selection, but Fylke level also possibleregions <- read.csv('geoid.csv');norway<-regions[which(regions$Country.Code == 'NO'& regions$Target.Type == 'County'& regions$Status == 'Active'),];

set.seed(1);

norway$isTreatment <- sample(c(0,1),nrow(norway), replace =T)

write.csv(norway, file='norway.csv');

# paste into AdWordswriteLines(as.vector(norway[which(norway$isTreatment == '1'),]$Canonical.Name),file('treatment.csv'));

Configuring Google AdWords I

Configuring Google AdWords II

Configuring Google AdWords III

Configuring Google AdWords IV

Configuring Google AdWords V

Done!!

Wait

. . . for the results

Google Analytics Core Reporting API & R

1. Create an “app”• Google Developers page• Enable Google Analytics API• Create Credentials: OAuth client ID, Application type: Other• Result: Client ID and Client Secret

2. Find your GA Profile ID

Setting up GNU R

client.id <- 'xxxxxxxxxxxxxxx.apps.googleusercontent.com';client.secret <- 'xxxxxxxxxxxxxxx';analyticsProfileId <- '111111111';

# redirect to google, paste, coderequire(RGoogleAnalytics);token <- Auth(client.id, client.secret)

# savesave(token, file = 'gatoken.txt');

# next timetoken <- load("./gatoken.txt")ValidateToken(token);

Create a query

query.list <- Init(start.date = "2015-10-01",end.date = "2016-02-29",dimensions = "ga:region,ga:date,ga:medium",metrics = "ga:sessions,ga:transactionRevenue",filter = "ga:country==Norway",max.results = 50000,sort = "-ga:date,ga:region",table.id = paste0("ga:",analyticsProfileId));

ga.query <- QueryBuilder(query.list);ga.data <- GetReportData(ga.query, token);

Real Data Example - www.flexponsive.net

kable(head(ga.data))

region date medium country sessions transactionRevenueBrussels 20160229 referral Belgium 1 0State of Parana 20160229 referral Brazil 1 0Baden-Wurttemberg 20160229 organic Germany 1 0Baden-Wurttemberg 20160229 referral Germany 1 0Rhineland-Palatinate 20160229 referral Germany 1 0(not set) 20160229 (none) Hong Kong 5 0

Tip: Query Explorer

Tip2: Dimensions & Metrics Explorer

Tip3: Avoiding sampling

> ga.data <- GetReportData(ga.query, token)Status of Query:The API returned 1393 resultsThe query response contains sampled data. It is based onXX.XX % of your visits. You can split the query day-wisein order to reduce the effect of sampling.

Set split_daywise = T in the GetReportData functionNote that split_daywise = T will automatically ....

• “Sampling occurs automatically when more than 500,000 sessions(25M for Premium) are collected for a report, allowing GoogleAnalytics to generate reports more quickly for those large data sets.”

Data Integration

• Wide Format: for each region and time a row• Long Format: Region / time / dimension one line (EAV)

require (reshape2);

## Loading required package: reshape2

w <- reshape (ga.data, timevar = 'medium',idvar = c( 'region', 'date'), direction = 'wide');

Data Integration: Almost finished

• Merge: Who is in which group?

ds <- merge (w, norway[, c ( 'Name', 'isTreatment')],by.x = 'region', by.y = 'Name', all.x = T)

• Data set is ready!• Comfortable DSL for data manipulation• Use packages to minimize code

Case Study: Wanderlust

Case Study: Wanderlust

• an app “developed” for this presentation• mysterious weekend getaway and short holidays booking engine• supports inventory management of hotels and airlines• seasonal demand fluctuations

Evaluation

• Simulated data for illustration: 3 summer months• 1st August: experiment starts in 10 random provinces (fylke) -AdWords stopped

• 1st August: start of school, search volume falls everywhere by 50%

• Scenario: 100% of visitors click organically when the AdWord invisible• Randomization has decided:

• Sor-Trondelag (Trondheim): In the treatment group - from 1st Augustno AdWords

• Hordaland (Bergen): In the control group - AdWords continue

Revenues in Sor-Trondelag (treatment)

60

80

100

120

Jun 01 Jun 15 Jul 01 Jul 15 Aug 01 Aug 15 Sep 01date

tran

sact

ionR

even

ue.to

tal

Revenues in Hordaland (control)

60

80

100

120

Jun 01 Jun 15 Jul 01 Jul 15 Aug 01 Aug 15 Sep 01date

tran

sact

ionR

even

ue.to

tal

Revenues in both Fylke

60

80

100

120

Jun 01 Jun 15 Jul 01 Jul 15 Aug 01 Aug 15 Sep 01date

tran

sact

ionR

even

ue.to

tal

region Hordaland Sor−Trondelag

ROI Calculation - standard regression

require(stargazer);out <- lm(transactionRevenue.total ~ isTreatment.cpc,

data = sd.w)stargazer(out, header=FALSE, type='latex')

Table 2

Dependent variable:transactionRevenue.total

isTreatment.cpc −48.358∗∗∗

(1.350)

Constant 111.350∗∗∗

(0.569)

Observations 1,748R2 0.424Adjusted R2 0.423Residual Std. Error 21.560 (df = 1746)F Statistic 1,282.996∗∗∗ (df = 1; 1746)

Note: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01

ROI Calculation - standard regression

• Standard OLS regression with binary variable == comparing means• But not the right ones. In this case:

Revenues = β0 + β1 ∗ treatment

• The treatment takes value 1 for the treatment group after theAdWords were stopped in Sor-Trondelag, otherwise 0

• As a result β1 represents the difference between the average revenuesin Sor-Trondelag in August and average revenues in Hordaland andSor-Trondelag in June and July

• That’s clearly now what we are looking for!!

Difference in Differences

ROI Calculation - Differences in Differences

require(plm)out <- plm(transactionRevenue.total ~ isTreatment.cpc,

data=sd.w, index=c("region", "date"), model="between")stargazer(out, header=FALSE, type='latex')

Table 3

Dependent variable:transactionRevenue.total

isTreatment.cpc 0.189(0.254)

Constant 102.741∗∗∗

(0.062)

Observations 19R2 0.032Adjusted R2 0.028F Statistic 0.556 (df = 1; 17)

Note: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01

ROI Calculation - Difference in Differences

• Difference in Differences estimator using fixed effects model withbinary varaibles allows to calculate the true effect of the treatment

• Econometrically we estimate this equation:

Revenues = β0 + β1 ∗ treatment + β2 ∗ before + γ ∗ fylke

• fylke is a matrix of binary variables for each district• before is a binary variable takes value 0 in a period in which AdWordswere running in all districts and value 1 in period in which experimentwas started in some regions

• treatment takes value 1 for the treatment group in the preiod inwhich the experimetn was started, i.e. after the AdWords werestopped in Sor-Trondelag, otherwise 0

• The estimation result reveals the true impact of AdWords onrevenues in this data set

Discussion

• The Missing counterfactual - we do not know what else could behappening - help: Experiment

• Challenge: Big Data without Big Code - Google Analytics & GNU R -Very rich toolbox

• Result: Differences in Differences can work - note assumptions

Table of Contents

Intro

Brand KeywordsThe eBay StudyCalculating the true ROI

Brand keywords with RConfiguring ExperimentUsing Google Analytics API

AdWords experiment: an exampleRegression Results