survey design: introduction & overview

Lecture 9 Week 14

Survey Design:
Introduction & Overview

Dr. James Neill
Centre for Applied Psychology
University of Canberra
28 September, 2009

Survey Design: Introduction & OverviewJames T. NeillCentre for Applied PsychologyUniversity of Canberra, ACT, Australia

The purpose of this presentation is to describe the nuts and bolts of designing a survey for research in the social sciences.

This presentation is partly based on some slides by Dr. Brent Ritchie currently at the School of Tourism at The University of Queensland (adapted with permission) e.g., the examples of survey questions about tourism in Canberra. Dr Brent Ritchie, Image sources: Background image: Workbench melee: http://www.flickr.com/photos/flattop341/1086598688/

Further info: http://en.wikiversity.org/wiki/Survey_design/Introduction_and_overview

Outline

Research process

Questionnaire design

Types of questions

Response formats

Sampling

Image sources:Questionnaires are by James Neill (License: Public domain), based on the Life Effectiveness Questionnaire

Objectives

To introduce key aspects of questionnaire design including question style, response formats, layout, and pilot testing

To consider implementation issues (such as sampling)

To demonstrate the importance of rigour in planning, developing, and implementing research questionnaires

Understanding common methods for the design and implementation of survey-based research and the pros and cons of each method (e.g., f2f interview, mail survey, internet survey). Understanding research design and implementation issues to be considered in survey research design Understanding the importance of a rigorous, step-by-step process for the development of instrumentation

Resources

Survey Design (Wikiversity)

Readings (Wikiversity)

Books

Look for books about surveys design and survey research in libraries.

Image: James Neill, from Flickr, cc-by-a

Research process

What is a survey?

A standardised stimulusA measuring instrument

Image soruces: Clipart (Art Explosion)

What is a survey?

A way of converting
fuzzy stuff
into hard data for analysis

Image source: DSCF3351 by joelogonLicense: Creative Commons Attribution 2.0

Purposes of survey research

Information gathering & describinge.g., polls, attitudes, demographics

Theory-building & testingExplanatory, e.g., why?

Predictive, e.g., what is likely to happen?

Often survey research does some of both.

Research purposes

DescriptiveCollects basic descriptive data/statistics e.g., consumer profiles(age, gender)

ExplanatoryExamine underlying data patterns

Linked to a hypothesis/research objective

Research purposes

PredictiveWhat happens if

Useful for marketing or assessing consumer behavior

Honours-MA-Ph.D survey research

Types ofQuestionnairesSelf -administeredInterview -administeredPostalquestionnaireDelivery andcollectionquestionnaireTelephonesurveyFace to facestructuredinterview

Web-based

Designing a survey

Image source: Questionnaire by TuppusLicense: Creative Commons Attribution 2.0

1. Formulate Generic Questionnaire2. Expand the QuestionnaireBased on study objectives

Turn intoseparatesections Question styles& types

3. Finalise Questionnaire-Pre-test/pilot test-Several drafts needed

Placement & Funnel Qs

Questionnaire planning/design

Formulate generic questionnaire

Turn objectives into sections of the survey

Ensure all questions relate to research objectives

For explanatory objectives or hypotheses ensure both dependent and independent variables exist

Cover letter / Ethics statement

Outline details of research projectPurpose

What's involved?

Explain any risks/costs/rewards

Contact details

Human Ethics approval #

How is consent given/not give?

How to return?

Can choose not to continue anytime

Instructions

Provides consistency - helps to ensure standard conditions across different administrations

Explain how to do the survey in a user-friendly manner

Example:
Life Effectiveness Questionnaire

Expanding the survey

Image source: http://www.flickr.com/photos/peretzpup/3059447579/

Screening

Does the participant qualify for the survey? (esp. for internet surveys)

Ask screening questions first, rather than later

Use branching if there are conditional questions

Flow and structure

Logical order of questions
(use sections)

Use funnel questions to move respondents through survey

Start off with easy to answer and engaging questions

More controversial questions in middle section

Personal questions start or end?

Survey design principles
Jenkins and Dillman (1995)

1. Use the visual elements of brightness, color, shape, and location in a consistent manner to define the desired navigational path for respondents to follow when answering the questionnaire.


2. When established format conventions are changed in the midst of a questionnaire use prominent visual guides to redirect respondents.


3. Place directions where they are to be used and where they can be seen.4. Present information in a manner that does not require respondents to connect information from separate locations in order to comprehend it.

Types of questions

Imag sourcese: http://commons.wikimedia.org/wiki/File:Aiga_information_.svghttp://www.flickr.com/photos/laffy4k/404298099/

Types of questions

Open-
endedClosed-
endedBe able to justify and defend your choices...

Image: http://commons.wikimedia.org/wiki/File:Aiga_information_.svg

Open-ended questions

Rich information can be gathered

Useful for descriptive, exploratory work

Difficult and subjective to analyse

Time consuming

Open-ended question examples

What are the main issues you are currently facing in your life?

How many hours did you spend studying this week? _________

Closed-ended questions

Important information may be lost forever

Useful for hypothesis testing

Easy and objective to analyse

Time-efficient

Closed-ended question types

Dichotomous questions

Multichotomous questions

The list (multiple response)

Ranking

Likert Scale

Graphical Scale

Semantic Differential

Non-verbal (Idiographic)

Dichotomous

Simple Yes / No response e.g.,

Excluding this trip, have you visited Canberra in the previous five years?__ Yes __ No

Multichotomous

Choice one of several answers e.g.How many hours did you spend studying this week?__ less than 5 hours__ > 5 to 10 hours__ > 10 to 20 hours__ more than 20 hours

eg. Which of the following statements best describes your reasons for taking a holiday to Canberra? (please tick one only) to visit friends and relatives for business for educational purposes for holiday/ sightseeing

Frequency scale

Over the past month, how often have you argued with your intimate partner?1. All the time2. Fairly often3. Occasionally4. Never5. Doesnt apply to me at the moment

Consider number of points (avoid over ~10)Consider directionConsider layout

The list (Multiple response)

Provides a list of answers for respondents to choose from e.g.,Tick any words or phrases that describe your perception of Canberra as a travel destination:__ Exciting __ Important__ Boring __ Enjoyable__ Interesting __ Historical

Ranking

Helps to measure the relative importance of several itemsRank the importance of these reasons for taking a holiday to Canberra (from 1 (most) to 4 (least)):__ to visit friends and relatives__ for business__ for educational purposes__ for holiday/ sightseeing

Likert Scale

Assesses differences of perceptions and allows measurement and comparison of these differencesIndicate your degree of agreement with this statement:I am an adventurous person.
(circle the best response for you)


Graphical rating scale

How would you rate your enjoyment of the movie you just saw?
Mark with a cross (X)

not enjoyable very enjoyable


Semantic differential

What is your view of smoking? Tick to show your opinion.

Bad ___:___:___:___:___:___:___ GoodStrong ___:___:___:___:___:___:___ WeakMasculine ___:___:___:___:___:___:___ FeminineUnattractive ___:___:___:___:___:___:___ AttractivePassive ___:___:___:___:___:___:___ Active


Non-verbal (Idiographic) Scale

Point to the face that shows how you feel about what happened to the toy.


Sensitivity & reliability

Scale should be sensitive yet reliable.

Watch out for too few or too many options

General aim:
Maximise sensitivity (i.e. more options)Maximise reliability (i.e. less options)

How many measurement options?Minimum= 2

Average= 3 to 7

Maximum= 10?

Scale of measurement guidelines

FEELING ABOUT SOMETHINGEXTREMELY POSITIVE EXTREMELY NEGATIVE 2-CategoriesGOOD NOT GOOD

3-CategoriesGOOD FAIRPOOR

4-CategoriesVERY GOOD GOODFAIRPOOR

5-CategoriesEXCELLENT VERY GOOD GOOD FAIR POOR

Watch out for too many or too few responses

Capital punishment should be reintroduced for serious crimes

1 = Agree2 = Disagree

1 = Very, Very Strongly Agree 7 = Slightly Disagree2 = Very Strongly Agree 8 = Disagree3 = Strongly Agree 9 = Strongly Disagree4 = Agree 10 = V. Strongly Disagree5 = Slightly Agree 11 = V, V Strongly Disagree6 = Neutral

Wording questions

Does the question focus directly on the issue or topic to be measured?
(If not, rewrite.)

Is the question stated as briefly as it can be?
(If too long, restate it more briefly.)

Alreck and Settle 1995

Wording questions

Is the question expressed as clearly and simply as it can be? I
(f the meaning wont be clear to every respondent, restructure the question.)

Use only core vocabulary - words and phrases people use in casual speech


Wording questions

Limit the vocabulary so the least sophisticated respondent would be familiar with the words

Use simple sentences where possible and complex sentences only when actually required

Use two or more short, simple sentences rather than one compound or complex sentences


Finalise questionnaire draft

LengthTry to keep them as short as possible

Only ask questions that relate to objectives

Tricks? Font size/double sided photocopying/numbering sections

Pre-testing and pilot testing

Pre-test try out on convenient others & revise

Pilot test try out on a small sample from the target population & revise

Be assertive and interactive about seeking feedback ask questions & observe

The customer is always right.

Maximising response rate

Layout and design is key

Respondents level of interest

Colour of paper

Accompanying letter / introduction

Mail surveys - self-addressed stamped return envelope

Rewards

Reminders or follow up calls

Examples

Examine the examples

What is wrong with the questions, if anything?

Example 1

How old are you?
___ 18-20
___ 20-22
___ 22-30
___ 30 and over

Example 2

Are you satisfied with your marriage and your job?__________________________

Example 3

You didnt think the food was very good, did you?_____ Yes _____ No

Example 4

Environmental issues have become increasingly important in choosing hotels. Are environmental considerations an important factor when deciding on your choice of hotel accommodation?
____ Yes ____ No

Example 5

What information sources did you use to locate your restaurant for todays meal?
(please tick appropriate spaces)
____ Yellow pages
____ Internet
____ Word of mouth

Pre-test & revise

Pre-test items and ask for feedback

Revise:

items which dont apply to everybody

redundancy

skewed response items

misinterpreted items

non-completed items

Reconsider ordering & layout

Survey format checklist

Introduction/covering letter or verbal introducatione.g. Who are you? Are you bona fide? Purpose of survey? Ethical approval? How results will be used? Confidentiality? Further info? Complaints?

InstructionsSets the mind frame, but be aware few people will read it without good prompting and being easy-to-read

Group like questions together

Consider order effects, habituation, fatigue, switching between response formats

Survey format

Font type / size, number of pages, margins, double vs. single-siding, colour, etc.

Demographics - usually beginning or end; only use relevant questions

Space for comments?

Ending say thanks!

Pre-test & revise/refine

Implementing surveys

Image source: http://www.flickr.com/photos/peretzpup/3059447579/

Comparison of
Data Collection Methods

Alreck and Settle (1995:32)

Sampling

Image source: http://commons.wikimedia.org/wiki/File:Marbles_canicas.PNG

Sampling terminology

Population

Sampling Frame

Sample

Representativeness

Population - set of all individuals having some common characteristic, e.g., Australians

Sampling Frame subset of the population from which the sample is actually drawn e.g., White pages

Sample set of people who actually contribute data to e.g., Every 1000th person in the white pages who answers the phone and responds

Representativeness How similar is the sample to the population with regard to the constructs of interest?

Why sample?

Why sampling rather than a census?

Sampling reduces:Cost, time, sample size and defines the research

If the sample is representative, allows inferences to be drawn concerning the total population

What is sampling?

Sampling is the process of selecting units (e.g., people, organizations) from a population of interest so that by studying the sample we may fairly generalize our results back to the population from which they were chosen.- Trochim, 2002

Sampling frame

The set of participants from which the sample is drawn

Examples:Electoral Rolls

Membership Lists (organisations, graduates association)

Telephone Book

Members of Specific Groups or Clubs (Fishing, Ramblers)

Households or post codes

Representativeness of sample depends on:

Adequacy of sampling frame

Selection strategy

Adequacy of sample size

Response rate both the % & representativeness of people in sample who actually complete survey

Note: It is better to have a small, good sample than a large, poor sample.

Sampling example:

Shere Hite
American Sexology

Male-female relations

Shere Hite doyenne of sex polls

Media furors & worldwide attention

127-item questionnaire about marriage & relations between sexes

4500 USA women, 14 to 85 years

Society and men need to change to improve lives of women

Some of Hites findings....

70% married for 5 years having affairs...

(usually more for emotional closeness than sex)76% did not feel guilty

87% had a closer female friend than husband

98% wanted basic changes to love relationships

only 13% married for 2+years were still in love

84% were emotionally unsatisfied

95% reported emotional & psychological harassment from their men

Some of the critical comments....

She goes in with prejudice & comes out with a statistic.

The survey often seems merely to provide an occasion for the authors own male-bashing diatribes.

Hite uses statistics to bolster her opinion that American women are justifiably fed up with American men.

Response rate & Selection bias - 1

100,000 questionnairesSent to a variety of womens groups
- feminist organisations, church groups, garden clubs, etc.4,500 replied
(4.5% return rate)

We get pretty nervous if respondents in our survey go under 70%. Respondents to surveys differ from nonrespondents in one important way: they go to the trouble of filling out what in this case was a very long, complicated, and personal questionnaire.
- Regina Herzog, University of Michigan Institute for Social Research

Response rate & Selection bias - 2

To learn more about Shere Hites research, visit her website: http://www.hite-research.com/

Sampling techniques

Probability (random) sampling Random

Systematic

ClusterMulti-Stage Cluster

Non-probability samplingQuota

Convenience

Snowball

Probability sampling - each member of population has a specific probability of being chosen.Random Sampling - everyone in population has an equal chance of being selected.

Systematic Sampling - e.g., every 10th student ID number

Stratified Random Sampling - population divided into strata, then random sampling from within each stratum (e.g., an equal number of males/females are selected)

Cluster Sampling - identify clusters of individuals & sample from these (e.g., 1 person per household)Multi-Stage Cluster Sampling (e.g., 1 person per selected household per selected suburb)

Non-probability sampling - arbitrary, sample not representative of populationQuota Sampling - e.g., 50% psychology students, 30% economics students, 20% law students

Convenience Sampling - take them where you find them method e.g., at shopping mall

Snowball Sampling - ask each respondent if they know someone else suitable for survey e.g., studying drug-users.

Random/probability sampling

Each unit has an equal (and perhaps known) chance or probability of selection in the sample

Selection occurs entirely by random chance

Often called representative sampling

Simple random sampling

Everyone in the target population has an equal chance of selection

Useful if clear study area or population is identified

Similar to a lottery

List of names are assigned #s and randomly select #s of respondents

Randomly select # through table of random #s or by computer

Systematic random sampling

Selecting without first numbering

Respondents (units) selected from a list/file.

Useful when survey population is similar e.g. List of Students, List of Package Tourists

Select sample at regular intervals from the population e.g., every 5th person on a list

Sometimes called file sampling

Systematic random sampling

Cannot do 1 in every 5

As then 4 people out of 5 stand no chance of being selected

Select a random starting point between 1 and 5

Sometimes called file sampling

Stratified random sampling

Sub-divide population into strata (e.g., by gender, age, or location)

Then random selection from within each stratum

Improves representativeness

e.g., Telephone interviews using post-code strata

Non-random/Non-probability

Also called purposive or judgemental sampling

Useful for exploratory research and case study research

Able to get large sample size quickly and useful when cant find a sample frame

Non-random/Non-probability

Make assumptions and maybe generalisations from your data, but not on statistical grounds

Limitations include potential bias and applicability

Convenience sampling

Sampling is by convenience rather than randomly

Due to time/financial constraints

e.g. surveying all those at a tourist attraction over one weekend

Purposive sampling

Respondents selected for a particular purpose e.g., because they may be typical respondents

e.g., select sample of tourists aged 40-60 as this is the typical age group of visitors to Canberra

e.g., Frequent flyers to contact regarding service quality in an airline setting

Snowballing

Useful for difficult to access populations e.g., illegal immigratnts, drug users

Respondents recommend other respondents

e.g., in studying ecstasy users, gain trust of a few potential respondents and ask them to recommend the researcher to other potential respondents

Summary of sampling strategy

Identify target population and sampling frame

Selection sampling method

Calculate required sample size

Maximise return rate

Task

A research project's aim is
To identify the behaviour and attitudes of UC students with regard to its computing services.What is the research population?

How might you get hold of a sample frame?

What sampling technique would you use?

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline LevelNinth Outline Level

Click to edit the title text format

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline LevelNinth Outline Level

survey design: introduction & overview

Education