Improved Questionnaire Design Yields Better Data: Experiences from the UK’s Annual Survey of Hours and Earnings

Jacqui Jones, Pete Brodie, Sarah Williams & Jane CarterUK Office for National Statistics

Overview

• Data Collection Methodology at ONS

• Methods standards

• Design standards

• Case study: The Annual Survey of Hours and Earnings

Slide 1

Background to DCM at ONS

• Mid 1990s 1st QTU set-up for Census

• Then expanded to include social surveys and business surveys

Slide 2

Methods Standards

• Aim was to develop a framework of processes to be used for all DCM question(naire) development work

• Outcome was the Data Collection Methodology Improvement and Measurement Framework

(DCM IMF)

Slide 3

The DCM IMF

Identification of data requirements

+Expert Review +

Evaluation Review

Stage 2 Redesign & Development

Stage 3 Pre-field Testing

Stage 4

Stage 1

Stage 5 Field Implementation

Identify and Disseminate Best Practice

Stage 7

Stage 6

Field Testing

Post-implementation Evaluation

Slide 4

The DCM IMF

Stage 1

Stage 2

Stage 3

Stage 4

Literature review

Expert review

Evaluation review

Instrument development

Expert review

Qualitative testing

Expert review

Re-development

Quantitative analysis

Qualitative follow-up

Slide 5

Design Standards

• Aim was to develop tested standards for the design of questions and questionnaires

• Outcome is draft standards

– with constraints – colour and cross boxes

Slide 6

Example of Design Standards

Example of Design Standards

Case Study: ASHE

• ASHE collects data for earning statistics e.g. minimum wage

• ASHE questionnaires are sent out to employers to complete on behalf of selected employees

• 250,000 sample of employees

Slide 9

ASHE: Need for Change

• National Statistics Quality Review

• New data requirements e.g. Pay for other reasons Pension data

• Existing data problems e.g.

Possible inclusion of pay elements in the wrong responses

Slide 10

Overview of the ‘Old’ Questionnaire

• Very cramped 2 sided questionnaire which collected 26 data items

• 4 sides of additional notes

• Respondent coding

Slide 11

‘Old’ Questionnaire

Slide 12

‘Old’ Questionnaire

Slide 14

Overview of the ‘Old’ Questionnaire

• Unclear data requirements

• Inconsistent terminology

• Double-barrelled questions

Slide 15

‘Old’ Questionnaire

ASHE Development

Followed DCM IMF processes:

• Focus group (8 large businesses)

• Pre-field testing• 30 cognitive interviews (5 waves of testing)

Waves 1-3 concurrent interviewingWaves 4-5 retrospective interviewing

Slide 17

ASHE Development

2004 Field tested 6 sided questionnaire:

• Split sample design (98% ‘old’ , 2% ‘new’)

• Quantitative analysisNon-responseRespondent burden (actual & perceived)Processing errors

Slide 18

Results of ASHE Field Test

• Response (no statistical difference)• Errors (no statistical difference)

• Respondent feedback (positive)• Respondent burden (reduced per question)• Some problems with question wording

• Costs (increased paper, printing, postage, processing)• Time (increased processing time)

Slide 19

6 sided to 4 sided compromises

• Minimum standard – based on testing - keep question & response category wording + instructions to stay with questions

• Challenge was to promote positive respondent perceptions and clear navigational path

• No time for further testing – redesign by DCM experts & ‘quick perception’ telephone interviews

Slide 20

‘Old’ Questionnaire

Slide 21

ASHE 6 Sided Questionnaire

ASHE 4 Sided Questionnaire

2006 ASHE 4 Sided Questionnaire

Slide 24

2005 Qualitative Follow-Up

• Looked at differences between responses to the 2004 ‘old’ questionnaire and the 2005 4 sided questionnaire.

• Focussed on 5 key areas e.g. the affect on basic pay of a new ‘other pay’ question

• 31 in-depth telephone interviews

Slide 25

‘Old’ Basic Pay Question

Slide 26

‘New’ Basic Pay and Other Pay Questions

Slide 27

‘Other Pay’ Results

Qualitative study found:• Better comprehension of new questions• Improved reporting of ‘other pay’• Respondents preferred ‘new’ layout

Quantitative analysis found:• ‘Old’ questionnaire - ‘other pay’ had been excluded

by some• Correction by imputation added 0.8% growth to

basic pay at the all employee levelSlide 28

What did we learn?

Must have:

• clear & agreed data requirements• agreement on the maximum number of pages• DCM experts involved in redesign & testing• method standards• design standards • improved methods for measuring data quality

Slide 29

Thank You

Jacqui.Jones@ons.gov.uk