What is an Opinion Survey or Poll?

GV917

Opinion Surveys and Polls

An Opinion Survey is a conversation between people made possible by a relationship of trust between them

This relationship differs in different survey modes, but it is always a ‘conversation’ and so subject to the complexities of human interaction

There are different types of opinion surveys and they all have advantages and disadvantages

Factors influencing responses to surveys

Relationships between interviewer and interviewee (interviewer effects)

The type of survey being done (mode effects) The type of questions asked: factual, attitudinal

– question wording and balance are important The subject matter of the survey – some things

are easy to ask about, others much less so The order of the questions in a survey The type of responses sought – closed or open

questions

An illustration of Complexities of Survey Questions: A Simple Factual Question from

the European Social Survey 2002F30CARD 56 Using this card, if you add up the income from all sources, which

letter describes your household's total net income? If you don't know the exact figure, please give an estimate. Use the part of the card that you know best: weekly, monthly or annual income.

J 01R 02C 03M 04F 05S 06K 07P 08D 09

H 10U 11N 12

(Refused) 77(Don’t know) 88

A Simple Factual Question from the European Social Survey 2002 - (look

at Non-Response)hinctnt Household's total net income, all sources

713 1.7 2.1 2.1

1752 4.1 5.3 7.4

2762 6.5 8.3 15.7

4722 11.1 14.2 29.9

4736 11.2 14.2 44.2

4113 9.7 12.4 56.5

3738 8.8 11.2 67.8

3136 7.4 9.4 77.2

4719 11.1 14.2 91.4

1978 4.7 5.9 97.4

554 1.3 1.7 99.0

326 .8 1.0 100.0

33248 78.5 100.0

4876 11.5

3573 8.4

660 1.6

9110 21.5

42358 100.0

1 J

2 R

3 C

4 M

5 F

6 S

7 K

8 P

9 D

10 H

11 U

12 N

Total

Valid

77 Refusal

88 Don't know

99 No answer

Total

Missing

Total

Frequency Percent Valid PercentCumulative

Percent

A Simple Attitudinal Question from the European Social Survey 2002 How interested are you in Politics? are you (READ OUT)

polintr How interested in politics

4614 10.9 10.9 10.9

15848 37.4 37.5 48.5

14487 34.2 34.3 82.8

7271 17.2 17.2 100.0

42220 99.7 100.0

31 .1

86 .2

21 .1

138 .3

42358 100.0

1 Very interested

2 Quite interested

3 Hardly interested

4 Not at all interested

Total

Valid

7 Refusal

8 Don't know

9 No answer

Total

Missing

Total

Frequency Percent Valid PercentCumulative

Percent

The Interview ‘Conversation’

Public opinion is dynamic and varied Some people are well informed and give

comprehensive answers to questions; Others are not very well informed and give very sketchy answers

Some people really don’t have an opinion about some issues and so will say that they don’t know or alternatively answer randomly

We have to know what polls can do and what they can’t do

What can Opinion Surveys Do? Opinion surveys can measure facts about

individuals quite well– their social characteristics, backgrounds and experiences

They can measure people’s attitudes quite well in relation to straightforward issues concerning politics and society and their own behaviour

They can give us a representative picture of what the nation is thinking about key political issues like the future of European politics

If they are panels they can give us trend data about the way people think or behave over time

What Can’t Opinion Surveys Do? Opinion surveys are not good at getting in-

depth narratives from people such as their life histories or detailed accounts of their time use – these should use qualitative in-depth studies

They are not good at probing really complicated issues in which people need to think about things a lot before answering

They have a limited ability to predict what people will do or say in the future – some things are predictable but other things not

Question Wording Effects – Rules of Thumb for drafting questions Keep questions simple not complicated “Are you interested in politics?” NOT “Are you interested in Parliamentary politics as

opposed to wider issues like Global Warming or the Debt Crisis?”

Avoid ambiguity “Have you travelled by air in the last year?” NOT: ‘Do you ever travel by air and by car?”

Question Wording Effects – Rules of Thumb for drafting questions Be specific not vague ‘Do you think the Conservative Government is doing a

good job or a bad job?’ NOT ‘What do you think of the political situation right now?’

Avoid leading questions ‘How likely is that you will vote at the next election?’ NOT ‘What party are you going to vote for at the next

election?’

A Question Wording Experiment- the Case of Party Identification The traditional wording of the party identification question in election

studies was inherited from the American National Election Study of the early 1950s, and used by Butler and Stokes in the first British Election Study in 1963. It appears in all subsequent surveys

‘Generally speaking, do you think of yourself as Labour, Conservative, Liberal Democrat (Scottish Nationalist/Plaid Cymru) or what?

If respondents say ‘None’ it is followed by a supplementary question: ‘Do you generally think of yourself as a little closer to one of the parties

than the others?’

This is then followed up with: Would you call yourself very strong {Party}, fairly strong, or not very

strong?

Criticisms of the Traditional Partisanship Question It is too leading – it assumes people are

attached to a political party, when many may not be. This may have been OK in 1963 but not any more when fewer people are attached to political parties.

For analysis purposes responses to the supplementary question are often added to the responses to the first question – making the bias in analysis even bigger.

Revised Party Identification Sequence ‘Some people think of themselves as usually being a

supporter of one political party rather than another. Do you usually think of yourself as being a supporter of one particular party or not?’

IF YES ‘Which party is that?’

Note this is much less leading – giving respondents ‘permission’ to say that they don’t support a party

What Difference Does it Make?Response Category Traditional wording Revised Wording

None 17.1 47.8

Labour 35.9 26.9

Conservatives 26.1 18.3

Liberal Democrats 14.6 5.1

Scottish Nationalists 1.4 0.5

Plaid Cymru 0.4 0.3

Greens 0.6 0.5

UKIP 1.0 0.6

BNP 0.2 0.1

Others 1.9 0.4

Question Wording Revisited

Question wording is really important. If one wants to bias answers, then it is easily done by phrasing the question in a given way.

However, a biased question can still be useful. The traditional question has been used in every election study since 1963 and it will be included in the next study.

Why? Because it provides a time series.

The Decline of Partisanship in Britain, 1964 to 2005

1.4

1.5

1.6

1.7

1.8

1.9

2.0

2.1

2.2

2.3

Year

Mean Strength (0-3 Sca

le)

1964 2005

R2 = .90

6466

70

74F

74O

79

8387

92

97

01

05

<--Regression Line

Mode Effects

These are associated with different ways of administering a survey

Face-to-face or ‘in-person’ surveys Telephone surveys Internet surveys Postal Surveys There are advantages and disadvantages of

different modes

Face-to-Face Surveys - Advantages We can get national probability samples. This is

very important for inferring the characteristics of populations from sample

The interviewer can persuade individuals to participate and thus increase the response rate

The interviewer can respond to individuals who don’t understand a question and clarify or explain things

The interviewer can report on how the interview went – eg.was the respondent attentive or not?

Face-to-Face surveys - Disadvantages Face-to-Face surveys are very Expensive (e.g., $5500 per

voter in the US National Election Study in 2004) Because they are expensive the N’s are often small – 792

voters in 2004 ANES Slooooow Moving – they take an average of 6 weeks to

complete Limited Ability for Panels – that is interviewing the same

people on more than one occasion One Study Only! Data Mining, Pre-Test Biases, replication is

very difficult In the case of election studies there is limited or No Inter-

Election surveys

39.6

31.7

22.4

6.3

36.1

30.8

24.4

8.7

36.2

33.2

22.6

7.9

0

5

10

15

20

25

30

35

40

45

50

Labour Conservative Liberal Democrat Other Parties

Per

cent

age

In-Person Internet Actual Vote

MODE EFFECTS 1Reported Party Choice in Face-to-Face and Internet Post-Election Surveys and Actual Vote in Britain, 2005 General Election

Mode Effect II - Turnout

34.7

24.8

24.7

17.3

37.9

17.8

20.2

27.2

25.2

21.5

18.7

9.9

10.6

21.8

18.0

9.1

0 5 10 15 20 25 30 35 40

2004 NAES, RDD

2000 NAES, RDD

2004 ANES, IP, trad

2004 ANES, IP, rev

2002 ANES, RDD, trad

2002 ANES, RDD, rev

2000 ANES, IP, rev

2000 ANES, RDD, rev

2004 CES, RDD

2000 CES, RDD

2001 BES Campaign, RDD

1964-2001 BES, IP

2005 BES, IP

2005 BES, Campaign, Internet

2005 BES, CSES, Internet

2005 BSA, CSES, Paper

Percent Overreport

Advantages of Internet Surveys They are cheap – in the election study the

face-to-face survey costs up to ten time more than an internet survey

They are fast – one can get 5,000 respondents within 48 hours on the internet while this would take 6 weeks face-to-face

It is possible to do experiments (eg. Feedback to voters experiment)

BUT – at the moment they are not probability samples

Conclusions Opinion Surveys are the core methodology

for empirical social science and are essential for studying individuals

A lot of aggregate analysis – such as modelling the relationship between the economy and behaviour is based on national data such as GDP per capita and spending by consumers which is collected by surveys

But users of surveys have to know their limitations