Business Intelligence and Decision Modeling

Week 12 Campaign Management:Testing & Performance

Outline Testing and Control Groups

(Chap. 14) Campaign Performance Measurement

(Chap. 12) Analytics and Modeling Revisited

(Chap. 13)

About Testing Interactive marketing Vs traditional

advertising

Testing Vs market research

Testing Vs management resistance

Testing for Effectiveness Copy (communications) Execution (creative) Package bundles and offers Clickstreams (email and websites) Cross-channels

Topics in Testing Methodology Sample size Control group or benchmark Test groups set aside for time length Half-life …

Testing Methodology

Half-life

100%

50%

33%

25%

Response

Time

Testing Methodology:CI for a given sample P +/- CI or P +/- Z x SEp or P +/- Z √(pq/n)

Testing Methodology:CI for a given sample CI = +/- Z x √(pq/n) If p = .04, n = 10,200 and Z = 1.96

CI = 1.96 √(.04 x .96/ 10,200) CI = .0038 Thus UCL = .0438; LCL = .0362

Two-tail test!Note: CL confidence level (Z), CI confidence interval (+/-),

UCL and LCL Upper and Lower confidence limits

Two-tail Test for CL

.025 .025

Z = 1.96 for 95% CL

.04.04 .0438.0362

-.00368 .00368

+/- 2 SD

Testing Methodology:Sample size for a given CL If CI = Z x √(pq/n) Then CI = Z x √(pq/ √ n √n = Z x √(pq) / CI

If p = .04, Z = 1.96 and CI = .0038 √n = 1.96 √(.04 x .96)/ .0038 √n = 101 or n = 10,215Two-tail test!

One-tail test for CL

Testing Methodology:CI for a given sample CI = +/- Z x √(pq/n) If p = .04, n = 10,200 and Z = 1.64

CI = 1.64 √(.04 x .96/ 10,200) CI = .003182 Thus UCL = .043182

One-tail test!Note: CL confidence level (Z), CI confidence interval (+/-),

UCL and LCL Upper and Lower confidence limits

One-tail Test for Sample

.05

Z = 1.64 for 95% CL

.04.04 .043182

.00318

Testing Methodology:Sample size for a given CL If CI = Z x √(pq/n) Then CI = Z x √(pq/ √ n √n = Z x √(pq) / CI

If p = .04, Z = 1.64 and CI = .0038 √n = 1.64 √(.04 x .96)/ .0038 √n = 84.57 or n = 7,152 One-tail test!

Test against Control One single item variable: 2 Groups

SPSS DM Control Package Test SPSS Independent Sample T-test Also available on Excel

One single item variable: 3+ groups SPSS Compare means Oneway ANOVA

Testing Interactive Marketing Treatments Two or more variables: experimental

design Two-way ANOVA (Excel and SPSS)

Testing Treatments (2)

B1 B2 Total A

A1 (A1,B1) (A1,B2) (A1)

A2 (A2,B1) (A2,B2) (A2)

Total B (B1) (B2)

Copy Rational Emotional Total Ask

Ask High Income 1 Income 3 Ask High 1-3

Ask Low Income 2 Income 4 Ask Low 2-4

Total Copy Rational 1-2 Emotional 3-4

Testing Treatments (3)

Rational Tone

Emotional Tone Total Ask

Ask High 8 1 5

Ask Low 5 10 8

Total Tone 7 5.5

Campaign Performance(Traditional and online)

Interactive Marketing Campaign Effectiveness

CPM, RPM, OPM (Cost, Revenue and Orders per M or thousand)

CPO (Cost per order)

CPR (Cost per Response)

Cost Measures CPM =

(Cost / Volume) * 1000(19 000$ / 50 000) * 1000 = 380$

CPM media((Media + Insertion)/Circulation) * 1000((40 000$ + (10 * 1 000))/ 1 000 000) * 1000 = 60$

(Insertions are charged in CPM)

Response Measures Response in % = (Response / Quant.) * 100

RPM = (Response / Quant.)*1000

OPM = (Orders/Quant.)*1000)

Email and Web Analytics

e-commerce/emailsCampaign Effectiveness

Delivered

Click to open (CTO)

Unique click / Click through rate (CTR)

Cost per impression (CPI)

Paid per click (PPC)

Cost per action (CPA)

Cost per lead (CPL)

Clickstream Analysis

Search PreferenceEdit profile

WelcomeMain message

Download OfferDiscount Offer

Second subject/offer

Links/ Directory Viral information

Clickstream Analysis

Clickstream Analysis

Email PerformanceName Email 1 Email 2

Mailed 1,172,839 1,172,839 Delivered 1,129,444 1,138,827 Del% 96.30% 97.10%

Opened 242,830 268,763 Open% 21.50% 23.60%     Unique Click 44,681 54,559 CTR 18.40% 20.30%

Conversion 1,385 1,582 Conv. Rate 3.10% 2.90%     Tot. Sales $169,551 $190,498$ per Del $0.15 $0.17Avg. Order $122.42 $120.42

Mailing Costs $10,555.55 $10,555.55Cost per Open $0.043 $0.039Cost per Click $0.236 $0.193Cost per Sale $7.621 $6.672