Sampling for Internet Research

Carma Research ModuleJeff Stanton

May 15-17, 2008 Internet Data Collection Methods (Day 2-2)

Universe, Population, Frame

Universe: U.S. Residents, 18+

Population: Males, 18+

Sampling Frame

…

May 15-17, 2008 Internet Data Collection Methods (Day 2-3)

Universe, Population, Frame, Sample

Universe: U.S. Residents, 18+

Population: Males, 18+

Sampling Frame

…

Sample

Malhotra, N. K. (1999). Marketing research: An applied orientation. (International 3rd edition). London: Prentice Hall.

May 15-17, 2008 Internet Data Collection Methods (Day 2-4)

Multiple Sources of Error(http://www.idready.org/courses/2005/spring/survey_SamplingFrames.pdf)

May 15-17, 2008 Internet Data Collection Methods (Day 2-5)

Sampling for Internet-Based Studies

Final Sample

OpenRecruitment

How to find, buy, or

build this list?

Email Invitation

May 15-17, 2008 Internet Data Collection Methods (Day 2-6)

Probability vs. Non-Probability

Superior Statistical“Projectability”

RDD – May provide truly random sample

Intra-Org Census as Sampling Frame

Limited or no Statistical “Projectability”

Assumes triangulationis required

Generalizability basedin part on theory foundation

May 15-17, 2008 Internet Data Collection Methods (Day 2-7)

Two types of Non-Probability Sampling

•Non-purposive–Convenience sampling–Open recruitment–Student samples in place of actual population–Snowball or chain (can also be purposive)

•Purposive–Expert, Quota, Politically important–Heterogeneity, Extreme/deviant, maximum variation, intensity–Homogeneity, modal instance, critical case, criterion, –Theory-based, disconfirming

Patton, M. Q. (1990). Qualitative evaluation and research methods (2nd ed.). Newbury Park, CA: Sage Publications.

May 15-17, 2008 Internet Data Collection Methods (Day 2-8)

Internet Sources of Samples

•Non-purposive–Email harvesting–Snowball, chain emails, or IM-Buddy lists–Banner ads–Site-based polls–Site-based links–Class rosters–Social networking sites

•Purposive–Organizational membership lists–Listservs of interest groups or professional organizations–Purchased sample frames–Standing panels–Amazon MechTurk

Amazon Mechanical Turk

Some pundits have commented that you can pay $0.03 per survey question. Responses come quickly: more than one response per minute in some cases.

Mturk Has Built-in Survey Capability

Mturk Provides Quick Responses

Internet Data Collection Methods (Day 2-12)

Criticisms of Internet Samples

Internet users per 100 people

Digital Divide: respondents from different communities have differing access to necessary tools

Non-sampled respondents providing responses

Sampled respondents providing bogus responses due to lack of proctoring

Amorphous sampling frame (e.g., due to posting an open link)

May 15-17, 2008 Internet Data Collection Methods (Day 2-13)

A response to these problems:Mixed Mode Design

• Telephone response rates have dropped from the 70-80% in the mid-20th century to less than 40%; mail response rates have always been low; Internet response rates have declined

• Solution: Combine mail, phone, Internet, and other modes

• Use best available sampling for each mode

• Instrument design presents another challenge: Uni-mode or mode specific

May 15-17, 2008 Internet Data Collection Methods (Day 2-14)

Preferred Strategy: Mixed Mode Sampling

Start with Theory

Answer: Who,When, Why

Develop MultipleSample Sources and

Research Modes

Sampling Frame 1: Mail or Other Sample

Sampling Frame 2: Internet Sample

KnownPopulation

Values

Triangulate:Calibrate &

Compare