Chapter 12 Designing Good Samples

Doubting the Holocaust?

An opinion poll conducted in 1992 for the American Jewish Committee asked:

Does it seem possible or impossible to you that the Nazi extermination of the Jews never happened?

Doubting the Holocaust?

An opinion poll conducted in 1992 for the American Jewish Committee asked:

Does it seem possible or impossible to you that the Nazi extermination of the Jews never happened?

22% of the sample said “possible.”

Why do you think this was?

The question was later reworded:

Does it seem possible to you that the Nazi extermination of the Jews never happened, or do you feel certain that it happened?

The question was reworded:

Does it seem possible to you that the Nazi extermination of the Jews never happened, or do you feel certain that it happened?

Now, only 1% of the sample said “possible.”

Additional examples of survey questions

Homosexuals in the military?

Three versions of the survey question.

1. “Do you approve or disapprove of allowing openly homosexual men and women to serve in the armed forces of the United States?

Additional examples of survey questions

Homosexuals in the military?

Three versions of the survey question.

1. “Do you approve or disapprove of allowing openly homosexual men and women to serve in the armed forces of the United States?

47% strongly or somewhat disapproved, 45% strongly or somewhat approved.

Homosexuals in the military?

2. “Do you think that gays and lesbians should be banned from the military or not?”

Homosexuals in the military?

2. “Do you think that gays and lesbians should be banned from the military or not?”

37% said they should be banned,

57% said not.

Homosexuals in the military?

3. “Should President Clinton change military policy to allow gays in the military?”

Homosexuals in the military?

3. “Should President Clinton change military policy to allow gays in the military?”

53% said no,

35% said yes.

The following questions were posed in a

survey several years ago:

1. Have you ever heard of the Bradley-Nunn bill on veterans’ housing?

2. Have you heard of Midwestern Life magazine?

Though neither the bill nor the magazine has ever existed…

1. 53% of those polled said yes.

2. 25% of those polled said yes.

In 1998 the Washington Post interviewed 1020 randomly selected adults. Half were asked:

1. How do you think President Clinton is doing as president?

2. Is the country headed in the right direction, or is it seriously off track?

In 1998 the Washington Post interviewed 1020 randomly selected adults. Half were asked:

1. How do you think President Clinton is doing as president?

2. Is the country headed in the right direction, or is it seriously off track?

50% said on the right track

The other half were asked the same two questions, but in reverse order. This time the results were

40% said on the right track

To what would you attribute the difference of 10%?

What qualities of this US Census question make it a good question?

Do you have COMPLETE plumbing facilities in this house or apartment; that

is , (1) hot and cold piped water, (2) a flush toilet, and (3) a bathtub or shower?

( ) Yes, have all three faciities

( ) No

Population vs. Sample

Population: The group of all of the individuals we want information about.Population parameter: the true value of a population attribute

Sample: The part of the population that we actually examine in order to gather information.Sample statistic: an estimate, based on sample data, of a population parameter.

Methods of Collecting Data

1. Census: A study that obtains data from every member of a population.

2. Sample survey: A study that obtains data from a subset of a population in order to estimate population attributes.

3. Experiment: A controlled study in which the researcher attempts to understand cause-and-effect relationships.

The study is "controlled" in the sense that the researcher controls (1) how subjects are assigned to groups

and(2) which treatments each group receives.

4. Observational study. Like experiments, observational studies attempt to understand cause-and-effect relationships. Unlike experiments, the researcher is not able to control

(1) how subjects are assigned to groups

and/or (2) which treatments each group

receives.

Problems in Designing Samples

The national magazine the Literary Digest had successfully predicted every election since 1916.

For the 1936 election, the magazine mailed out ten million sample ballots to people whose names were taken from subscription lists of magazines, telephone directories, and automobile registrations. More than 2.3 million ballots were returned, the largest number of people to every respond to a survey.

The election prediction:

Alf Landon would defeat Franklin Roosevelt by a margin of 57% to 43%.

Roosevelt: 161 electoral votes

Landon: 370 electoral votes

Do you recall President Landon?

The election results!

Franklin Roosevelt won in a landslide victory with 62% of the popular vote.

Roosevelt: 523 electoral votes

Landon: 8 electoral votes

What went wrong?

Convenience Sampling

Choosing individuals who are easiest to reach.

Problematic because if often results in bias, usually do to undercoverage.

Bias: a systematic error in the data that favors certain outcomes.

Interviewing at the Mall

Manufacturers and advertisers often use interviews at shopping malls to obtain information about the habits of customers and the effectiveness of advertisements.

Do you think these surveys accurately describe what consumers think?

Why or why not?

Ann Landers asked parents the following question in her column:

If you had it to do all over again, would you have children?

70% of the over 10,000 people who wrote in said, “no, kids weren’t worth it.”

A more carefully designed survey later showed that 90% of parents are actually happy with their decision to have children.

Why do you think there was such a difference?

Nonresponse Bias

Bias resulting from respondents differing in meaningful ways from nonrespondents.

Online PollsThe American Family Association (AFA) is a conservative group which states it stands for “traditional family values.” It regularly posts online poll questions on its Website – just click on a response to take part.

Do you think their poll results usually support or oppose the AFA’s positions? Why?

Voluntary Response Bias

Bias occurring when sample members are self-selected volunteers.

Sampling Soup

If you are making a pot of soup. How would you decide if you need to add more spices?

Random Sampling is Best!

Random Sample: A sample in which

(1) The selection of a sample unit is based on chance.

(2) Every element of the population has a known, non-zero probability of being selected.

Simple Random Sample: A sample chosen from the population so that every individual in the population is equally likely to be chosen. (SRS)

How to Use a Random Number Table

1. Assign a numerical number to every individual in the population.

2. Choose a row from a random number table and choose the numbers for your sample.

NOTE: You must not skip over any numbers or rows as you use the table.

This is one row, scaled up, from a random number table.

8417706757 1761315582 5150681435 4105092031 0644905059 5988431180

Stratified Sample: A sample in which the population is divided into groups, based on some characteristic. Then, within each group, a SRS is selected. In this type of sample, the initial groups are called strata.

Cluster Sample: A sample in which every member of the population is assigned to one, and only one, group, called a cluster. A sample (usually a SRS) of clusters is chosen, and the members of those clusters only are surveyed.

Multistage Sample: A sample in which a combination of different sampling methods are used.

Stratified Sample: A sample in which a list of every member of the population is first made. From the list, the first sample element is chosen from the first k elements on the population list. Thereafter, every kth element on the list is chosen.