have a question you’d like to ask regarding today’s ... blending final.pdf · ask regarding...

Have a question you’d like to ask regarding today’s presentation?

We welcome you to typeyour questions in the ‘Question & Answer’ window at any time

during today’s Webinar. We will answer as many questions as time allows during the Q & A session

following this presentation.

www.peanutlabs.comTwitter: @PeanutLabsMR

To Blend or Not to Blend … That is the Question: A Sample Blending Ideation Session

Moderated by:

• Jeffrey Henning, Founder & VP, Strategy, Vovici

Presentations by:

• Steve Gittelman, President, MKTG Inc.

• Jackie Lorch, Vice President, Global Knowledge Management, Survey Sampling International

• John Bremer, Senior Vice President, Global Representativeness and Co-Director of the Harris Center for Innovation, Harris Interactive

Jeffrey Henning

• Co-founded Perseus Development Corporation in 1993

• Over twenty-two years of experience in the market research industry

• Developed SurveySolutions® for the Web, which won the PC Magazine Editors’ Choice Award for web-survey software

• Authored the eBook, Survey Software Success

• Entrepreneur, blogger and proud father of five

Today’s Agenda:

• How sample blend works within the MKTG, Inc. Grand Mean distribution methodology

• The necessity of sample blending and how to optimize sample blend

• How to monitor consistency and why it’s necessary

• How to incorporate sample blending techniques into established projects and reports to maintain consistency and validated results

Steve Gittelman

• Steve is president of Mktg, Inc. whose Sample Source Auditors have brought us the Grand Mean Project.

• He is an old school market researcher who believes that online quality is achievable and therefore an industry requirement.

• He is the author of over a hundred technical papers, with a number coming out this year.

• As a hobby Steve writes historical biographies and informs us that his book on the Vanderbilts is now available on Amazon.com

• Steve received his doctorate from the University of Connecticut.

• He joins us from Kuala Lumpur, Maylaysia where it is now 2:00 in the morning.

Presented by:Steve GittelmanMktg, Inc.

Players come in all sizes and shapes.

Baseball is a game of stats.Blending is a science of statistics.

• We can blend in two ways:

–The first is by understanding groups and the second is by understanding individuals.

• We measure groups through subsamples.

• In contrast we could measure all the individuals of a group and then blend them for each study as it comes along, one at a time.

We need to know how to keep score.

The standard could be a census or some other type of model like the Grand Mean.

Keeping score by segmentation.

• In our case, we use behavioral profiling.

• Buying behavior is a good example. In many cases it is what counts most for our clients.

• The distribution of behaviors across a population serves as a second standard, or model.

We look at the segmentations of different sources and blend them.

Here’s an example of a buyingbehavior blend with four sources.

Buyer Behavior OptimizationSample

Source (SS)

Broad

Range/Credit

Price sensitive

/

shoppers

Credit /

Environment

Domestic /

Coupons Optimization

SS#1 32% 16% 24% 29% 22%

SS#2 37% 14% 21% 29% 56%

SS#3 30% 17% 24% 29% 1%

SS#4 32% 15% 22% 30% 22%

US Grand

Mean 36% 14% 23% 27% 100%

Optimum

Blend 35% 15% 22% 29% *3.0%

*This

represents the

RMS error for

buying

behavior

segment alone

Optimum Blend on BBThese distributions are not significantly different but fall between one and two standard errors. The largest difference is a 9% relative change with the Domestic/Coupons segment. Note that

the average relative difference is 4.7%.

Distribution of the Buyer Behavior Segments

36%

15% 14%

22% 23%

29% 27%

34%

0%

20%

40%

60%

80%

100%

Preferred Optimum Blend

(DPI=T)

US Grand Mean

% o

f R

es

po

nd

en

ts

Broad Range/Credit Price Sensitive Shoppers

Credit/Environment Domestic/Coupons

A New Era• The best blends are where we accumulate data one

respondent at a time.

• Demographics and behavior could be used as two layers.

• Behavioral segmentations could classify individuals just like they do groups.

• Blends could be created for each study. One at a time.

• Each study would be the perfect blend.

• Respondents are carefully profiled.

……….a new era continued

• Think of being able to balance behavior as well as demographics.

• We could control for different levels of:– Respondent experience.– Purchasing intent.– Socio-graphic, media, and other behavior.– Make mid study corrections– Even look at non-response error.

The power is in the blends.

The best blend in the business…

• Each player chosen to round out the skill set.

• Each member of the team fully described by abundant and available statistics.

• A winning combination.

There is only one place where they do it like that….

.Putting together this team was no accident. It is the perfect blend.

Any questions for Steve?

We welcome you to type your questions in the ‘Question & Answer’ window at any time during

today’s Webinar. We will answer as many questions as time allows during the Q & A session following this

presentation.

Jackie Lorch

• Joined Survey Sampling International in 1990

• Was a member of the team which developed SSI’s first online panel

• Has authored and co-authored a number of papers addressing online data quality

• A native of Britain

• Received a BA in Greek Civilization and English at the University of Leeds in England and an MBA in Marketing from the University of Connecticut

Superior data wrapped in an engaging experience

Blending: necessary evil or game-changer?• Blending has traditionally been a last resort

• Blending increases heterogeneity

Superior data wrapped in an engaging experience

The world of communication is changing

• E-mail usage down 41% since 2003 Online Publishers Association

• 52% of consumers blog about their brands Razorfish

• Those spending 10 hours a week online, have cut use of traditional media by 65% Stanford University

Superior data wrapped in an engaging experience

Many people will never join a panel

Superior data wrapped in an engaging experience

Superior data wrapped in an engaging experience

Quotas work if stratification is relevant to questionnaire topic

Superior data wrapped in an engaging experience

Product may be liked across demographics

• So what drives the preference for Coke over Pepsi?

Superior data wrapped in an engaging experience

Beyond socio-demographics

Superior data wrapped in an engaging experience

Factors considered

• Personality traits • Need for cognition (Cacioppo et al)

• Music preferences (Rentfrow) • Neurographics measures

• Cognitive ability (Kahneman and Frederick)

• Propensity factors e.g. to participate, share, risk averse, attitude to privacy

• Geographic / personality alignment (Gosling and Rentfrow)

• Chronotypes, “lark” or “owl” (Roenneberg)

• Social Values. Schwarz has international benchmarks

• Disruption/orienting reflex measure, habituation disruption

• Broad set of factors which would define groups of people, spanning multiple disciplines

• Each test chosen as typical in class

Superior data wrapped in an engaging experience

Factors moved the needle

• 162 factors

• Tested against questions on technology, hobbies, interests, brand preference, loyalty, ad awareness

• Variables tested on power to “move the needle” on dependent variables

Superior data wrapped in an engaging experience

Findings: Test 1

1. Explained more variance than socio-demographics alone (20% v. 7%)

2. Neither cluster nor socio-demographics explained most of the variance

Do gains justify additional complication?

Superior data wrapped in an engaging experience

Step 2: Multi-source test

Superior data wrapped in an engaging experience

Test 2 results: Differences across sources

Sample sources*: A B C D E F SD Variance

Universalism as principle 5.75 5.49 6.02 5.90 5.80 5.52 1.72 0.044

Prayer in school, agree 2.92 3.03 2.50 2.84 3.07 2.93 1.44 0.042

Hedonism as principle 5.48 5.24 5.55 5.54 5.52 5.09 1.88 0.037

Tradition as principle 5.62 5.62 5.98 5.79 5.72 5.41 1.69 0.037

Trust internet transactions 2.93 2.62 3.11 2.86 2.84 3.03 1.29 0.029

See self as disorganized, careless 2.36 2.32 2.53 2.59 2.39 2.74 1.57 0.026

Traditional role of women 3.41 3.56 3.10 3.37 3.37 3.50 1.26 0.025

See self as critical, quarrelsome 3.11 3.12 2.83 3.11 3.07 3.32 1.64 0.025

Superior data wrapped in an engaging experience

Psychographics outperformed demographics

No balancing Balancing with socio-

demographic variables

Balancing with psychographic, etc. variables

Between sample variation metric 1*

0.88 0.76 0.48

Improvement factor 1.15 1.83

Between sample variation metric 2**

18% 14% 9%

Improvement factor 1.29 1.92

*This metric was calculated as follows: 1. Means of IVs tabulated per source; 2. Variance between sample means calculated; 3. Sum of variances calculated

**Calculated % of subsample means deviating 0.1 SD from total score

Superior data wrapped in an engaging experience

One example: Technology ownership

Willingness to try new things

Attitude

Superior data wrapped in an engaging experience

One example: Technology ownership

Ownership of BlackBerries, etc. varied by source

Ownership

Connects to level of adoption of new technologies

ActivityConnects to willingness to try new things

Attitude

Superior data wrapped in an engaging experience

Mixing demographics

Superior data wrapped in an engaging experience

Mixing psychographics

Superior data wrapped in an engaging experience

Mixing psychographics

Superior data wrapped in an engaging experience

Summary

• Testing calibration measures

• “State of the water”

Superior data wrapped in an engaging experience

In summary

• Multi-sourcing is the future

• We can monitor consistency

• Socio-demographics alone don’t do the job

Superior data wrapped in an engaging experience

Any questions for Jackie?

We welcome you to type your questions in the ‘Question & Answer’ window at any time during

today’s Webinar. We will answer as many questions as time allows during the Q & A session following this

presentation.

John Bremer

• Joined Harris Interactive in 1999

• Specialized in areas of non-probability sampling, rare populations, selection bias and online survey research

• Well known for having developed the proprietary propensity score weighting technique utilized by Harris, as well as producing successful election forecasts in the US in 2000-2008 and in the UK in 2005

• MS in Statistics from the University of Chicago and an MBA from the University of Chicago Booth School of Business

Sample Blending – When a Baseline Is Already Established

John Bremer

SVP, Harris Interactive

4/28/10

Background – Foundations of Quality

• Major finding of Foundations of Quality research is that different sample sources may produce different results even after the demographic characteristics are equalized.

– Results often overblown

– New ARF Research on Research Committee

• Implication: Switching between samples or different blending proportions could mean different results.

• Decision whether and how to blend depends on context– Beginning of tracker

– Stand-alone project

– Shift in sample composition

– Norms development

Motivating Problem – Going from Baseline to Blend

Baselines = Balance between Representativeness and Consistency

• Shift in sample composition when a baseline has been established creates unique issues

– Are differences due to shift in sample or real changes in marketplace? What if no differences observed when they should have been?

– Are comparisons to established norms resulting in correct decisions if not the same sample composition?

• Shifts might be for right reason but create trending issues– Better representativeness might create shift in trend line hurting ability to detect

change.

• Ad Testing

– Norms developed over time from a single sample source.

– Given volume of work and project requirements additional sample sources needed over time.

– Go/No-go decision criterion sensitive to small changes in results.

• Tracking

– Initial waves conducted with a particular and consistent blend of sample sources.

– Economic considerations lead to changing the sample composition.

Real World Examples

Basic Philosophy on Correcting Blended Samples to a Baseline

• Detect – Is there a difference?

• Calibrate – What would correct the difference?

• Estimate – Apply the correction.

• Validate – Check to see that things are how you think they should be.

A Generic Primer – Step 1

• Step 1: Identify potential variables

– Must be correlated with outcomes of interest that differ across samples.

– Must include demographics but go beyond demographics as well.

• Step 2: Run regression to determine which variables are likely to contribute to differences

– Dependent variable is indicator of which sample respondent comes from

– Independent variables are key variables from Step 1

• Step 3: Weight one sample to look like the other sample on variables found to be key.

• Step 4: Do it again until nothing new is found

Generic Primer – Step 2

• Step 1: Identify potential variables

– Must be correlated with outcomes of interest that differ across samples.

– Must include demographics but go beyond demographics as well.

• Step 2: Run regression to determine which variables are likely to contribute to differences

– Dependent variable is indicator of which sample respondent comes from

– Independent variables are key variables from Step 1

• Step 3: Weight one sample to look like the other sample on variables found to be key.

• Step 4: Do it again until nothing new is found

General Primer – Step 3 and Step 4

• Step 1: Identify potential variables

– Must be correlated with outcomes of interest that differ across samples.

– Must include demographics but go beyond demographics as well.

• Step 2: Run regression to determine which variables are likely to contribute to differences

– Dependent variable is indicator of which sample respondent comes from

– Independent variables are key variables from Step 1

• Step 3: Weight one sample to look like the other sample on variables found to be key.

• Step 4: Do it again until nothing new is found

Harris Interactive EPS Approach

• For each new sample, logistic regression performed. – Dependent variable is binary variable with 1 being the base sample, 0 otherwise.

– Independent variables are demographics and key variables.

– Probability of being in base sample is metric used to match samples

– Weighted to distribution of base sample probability

– Validate and repeat if necessary

– The samples are finally combined to represent the base population.

The EPS Approach in Pictures

Sample 1

Sample 2

BaseSample

EPS Weighted Sample 1

EPS Weighted Sample 2

Total Sample

BaseSample

Does it Work? Let’s Test It!!!

• Characteristics of the Experiment– Experimental Cells: Sample Supplier 1, Sample Supplier 2, Sample

Supplier 3

– Control Cells: Norm development sample

– N = 225 / Vendor / Execution (approximately)

– Conducted between 8/19/09-9/1/09

– Sample Balanced by Age Group, Education, Race, and EPS by Execution Cell

The Design

Table 1: Experimental Design

Control Cell Result TV Print

Successful

Acceptable

Not Successful

The Result

Supplier 1 Supplier 2 Supplier 3

Total Sample Size 1385 1338 1416

Weighting Efficiency 47% 51% 45%

Percent of Similar Business Decisions 83% 83% 67%

Reduction in bias

-Overall 40.4% 45.8% 44.7%

-Decision Criteria Questions 44.2% 51.1% 43.2%

-18-24 Year Olds 36.5% 55.3% 40.0%

Note: Reduction in bias is defined as percentage change in difference between base sample and supplier sample before and after adjustment

The Result – Yes I know it is the same, just needed more time

Supplier 1 Supplier 2 Supplier 3

Total Sample Size 1385 1338 1416

Weighting Efficiency 47% 51% 45%

Percent of Similar Business Decisions 83% 83% 67%

Reduction in bias

-Overall 40.4% 45.8% 44.7%

-Decision Criteria Questions 44.2% 51.1% 43.2%

-18-24 Year Olds 36.5% 55.3% 40.0%

Note: Reduction in bias is defined as percentage change in difference between base sample and supplier sample before and after adjustment

Blending the Result at the Aggregate Level – It Works!!!

Base Sample Proportion Control Cell Match %

0% 72.4%

10% 88.9%

20% 88.9%

30% 100.0%

40% 100.0%

50% 100.0%

60% 100.0%

70% 100.0%

80% 100.0%

90% 100.0%

100% 100.0%

Selectively Blending the Results – Works Even Better!!!

• 18-24 year olds is the limiting case. Would using no sample from the base sample in this age group result in different actions being taken? At what level would it not matter?

Percent 18-24 Sample from Base Sample Control Cell Match %

0% 94.4%

10% 94.4%

20% 100.0%

30% + 100.0%

Conclusions

• While F.O.Q research found differences across sample suppliers, the differences can be managed.

– Differences not big in many cases

– When baseline established, can still blend but must go through the DCEV Process

• Detect

• Calibrate

• Estimate

• Validate

– Must include demographics while also going beyond demographics for solution

• Demographics can be easily solved. Usually the answer is beyond demographics

– The method works!!!

Any questions for John?

We welcome you to type your questions in the ‘Question & Answer’ window at any time during

today’s Webinar. We will answer as many questions as time allows during the Q & A session following this

presentation.

Q & A Session

We welcome any questions you may have regarding the content of today’s Webinar.

Special thank you to each of our presenters!

Thank you for joining us!

The slide deck along with a recording of today’s presentation will be available for download via our

website. We will be sending all attendees a link to theslide deck as soon as it is available.