understanding society conference 25 july 2013 paul mathews knowledge, analysis and intelligence...

Understanding Society Conference25 July 2013

Paul Mathews

Knowledge, Analysis and Intelligence Directorate, HM Revenue and Customs Institute of Social and Economic Research, University of Essex

Question ordering effects on the reporting of fertility intentions and close social networks

Question ordering - Context Effects

Change in the answers to a survey questionnaire as a function of the previous items in the questionnaire’ Tourangeau et al, 2003

Examples Context Effects

Vodka or beer questions influences rating to how ‘Germanic’ is wine drinking? (Schwarz, Munkel and Hippler, 1990)

Life Satisfaction preceding Marriage Satisfaction r = 0.32, Marriage satisfaction preceding Life Satisfaction r = 0.67 (Schwarz, Strack and Mai 1991)

Frequency of Context Effects General Social Survey (US) batteries of questions rotated. Only

4% of questions effected by placement (Smith, 1988) Needs to be a conceptual link

Question ordering - Context Effects

Question priming bias as domain sampling

Particularly in multipurpose longitudinal research (Time series - Change over time? Changes in preceding questions?)

Plausible risk

Why are fertility intentions important?

“The changing face of London: A baby boom is sending the city’s planners back to the drawing board”

The Economist 28th Jan 2012

“By 2015-16 greater London will need around 70,000 more school places”

Measurement problems…

Uncertainty / ambivalence

Context dependent… Preferences change over time

Age, ageing, life course, cohort, period Is there a ‘correct’ age to measure FP?

Experience of children Partnership and partner’s preferences Competing preferences

economic, cultural, leisure etc…

Because fertility preferences are so context dependent, then will the context in the questionnaire matter i.e. preceding questions?

Millennium Cohort Study – Wave 1

“How long did the labour last?” “Which, if any, of the following

types of pain relief did you have at any time during labour?”

Before asking “Do you plan to

have any more children?”

Social networks

Numerous concepts and operationalisations Flows through social networks Social capital Strength of weak ties Relatedness

At risk of context effects? E.g. prime a domain such as ‘work’ or ‘family’… does this influence who is ‘in’ your social network

My empirical work

Mortality experiments

Randomised (systematically identical) groups. o Treatments: priming questions then fertility questionso Controls: fertility questions then priming questions

Adult (own) mortality priming questions • 11 Questions• “What age do you expect to be when you die?”

Data collected 2006 and 2008-09 Published - Mathews and Sear 2008

Students internet experiment

Results: Significant increase in MALE ideal numbers of children. No effect for females •

15

Why?

Not mutually exclusive… Fatigue? Negative mood? Old age support (in adult prime)? First item in battery of fertility preferences? (DHS ideal

question) Own mortality is a ‘shock’ to non-decision decision?

(Competing preferences, cultural output and sociological modernity)

Social Psychological - Terror Management Theory (TMT) social immortality?

Evolutionary biology – Life History Theory (perceived risky environment should alter reproductive strategy)?

Sheer chance?!• Replication

Innovation Panel experiment

Waves 4 and 5 of Innovation Panel sub sample of 1,500 households - NOT STUDENTS!

Randomisation at household level

Controls Wave 4: Experiment after mental wellbeing

“I've been able to make up my own mind about things”

5 point scale: [All of the time – None of the time] Wave 5: Experiment after GHQ

“Have you recently been feeling reasonably happy, all things considered?”

4 point scale 1 More so than usual 4 Much less than usual

Two question ordering ‘treatments’

Fertility Intentions: “Do you think you will have any (more) children?”

[1 Yes, 2 Self / partner currently pregnant, 3 No] if the answer is yes “How many (more) children do you

think you will have?”

Close social network (i.e. 3 closest friends) ‘Please choose the three people you consider to be your

closest friends... They should not include people who live with you but they can include relatives’

Sex, Age, relatedness, frequency of contact, how far away they live etc

‘Is this friend a relative?’ [ 1 Yes, 2 No]

19

Descriptive statistics

Observations 696 Wave 4 N=409, Wave 5 N=287

223 individuals measured twice (27 changed their minds on wanting children)

Background demographics remain very similar across waves - Male 60%, Age mean 37.5 (SD 13.4) median 39 (split dummies in model), Parents 48%, Employed 72% (11% full time students), Married 45%, Lives with a parent 22%, sibling 14%.

Fertility intentions27 close social

network questions

(Nine questions for three

friends)

Fertility intentions questions

wave 4 ‘Make mind’ or wave 5

‘happiness’

(No social network

questions)

Social network

Close social network questions

1 or 2 fertility intentions questions

wave 4 ‘Make mind’ or wave 5 ‘happiness’

(No fertility intentions questions)

Results – Fertility intentions

All participants - reporting expecting

a(nother) child

Just unmarried participants - same

Wave 4 Wave 5 Wave 4 Wave 5

Treatment –preceding close social network

34.4% 50%

Control 1 – make mind up question

27.0% 37.1%

Control 2 – general happiness question

/ /

Number of participants

409 225

P-value of a t-test between control and treatment within the wave (Note: without Bonferoni correction)

0.052 0.03

Results – Fertility intentions

All participants - reporting expecting

a(nother) child

Just unmarried participants - same

Wave 4 Wave 5 Wave 4 Wave 5

Treatment –preceding close social network

34.4% 32.7% 50% 48.8%


27.0% / 37.1% /


/ 33.6% / 47.4%

Number of participants

409 287 225 160


0.052 0.44 0.03 0.43

Results – Social network

All participants - reporting a relative in their close social

network

Wave 4 Wave 5

Treatment –preceding fertility intentions question

31.4% 26.5%


29% /


/ 25%

Number of participants 409 287


0.30 0.38

Conclusions

Fertility intentions at risk of preceding questions Plausible risk...

Little evidence relatedness (or any other characteristics) of their close social network at risk of preceding questions

Important to construct and read questionnaires as a whole

Repeated measures: Replication, replication, replication

Acknowledgements

Participants in all studies

Maria Iacovou, University of Essex

Rebecca Sear, London School of Hygiene and Tropical Medicine

Ernestina Coast London School of Economics and Political Science

UK Economic and Research Council for funding

UKHLS Methodological Advisory Committee for accepting proposal

ISER and HMRC secondment

*Advert* - HM Revenue & Customs Datalab

• Compliance• Corporation tax• Self assessment• Value added tax• Stamp duty land tax• Trade statistics • Tax credits • Tobacco

• Variable names and descriptions are available on our website:

• www.hmrc.gov.uk/datalab/data.htm

Conclusions

Fertility intentions at risk of preceding questions Plausible risk...

Little evidence relatedness (or any other characteristics) of their close social network at risk of preceding questions

Important to construct and read questionnaires as a whole

Repeated measures: Replication, replication, replication

understanding society conference 25 july 2013 paul mathews knowledge, analysis and intelligence...

Documents