summary and conclusion - survey research and design in psychology

Survey Design II

Lecture 10
Survey Research & Design in Psychology
James Neill, 2017
Creative Commons Attribution 4.0

Summary & Conclusion

Image source: http://commons.wikimedia.org/wiki/File:Pair_of_merops_apiaster_feeding_cropped.jpg

7126/6667 Survey Research & Design in PsychologySemester 1, 2017, University of Canberra, ACT, AustraliaJames T. Neill

Description: Summary and conclusion to the Survey research and design in psychology lecture series.

Home page: http://en.wikiversity.org/wiki/Survey_research_and_design_in_psychologyLecture page: http://en.wikiversity.org/wiki/Survey_research_and_design_in_psychology/Lectures/Summary_%26_conclusion

Image name: Pair of merops apiaster feeding cropped.jpgImage source: http://commons.wikimedia.org/wiki/File:Pair_of_merops_apiaster_feeding_cropped.jpgImage author: Pierre Dalous, http://commons.wikimedia.org/wiki/User:Kookaburra_81License: CC-by-SA, http://creativecommons.org/licenses/by-sa/3.0/deed.en

Overview

Module 1: Survey research and designSurvey research

Survey design

Module 2: Univariate and bivariateDescriptives & graphing

Correlation

Module 3: PsychometricsExploratory factor analysis

Psychometric instrument development

Module 4: Multiple linear regressionMLR I

MLR II

Module 5: Power & summaryPower & effect sizes

Summary and conclusion

Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domainLast chance to ask questions along the way as we review what the unit has coveredNo such thing as a silly question

Survey research
(Lecture 1)

Types of research

Surveys are used in all types of research: Experimental

Quasi-experimental

Non-experimental

What is a survey?

What is a survey?A standardised stimulus for converting fuzzy psychological phenomenon into hard data.

HistorySurvey research has developed into a popular social science research method since the 1920s.

Purposes of research

Purposes/goals of research:Information gatheringExploratory

Descriptive

Theory testing/buildingExplanatory

Predictive

Survey research

Survey researchPros include:Ecological validity

Cost efficiency

Can obtain lots of data

Cons include:Low compliance

Reliance on self-report

Survey design
(Lecture 2)

Self-administeredPros: cost

demand characteristics

access to representative sample

anonymity

Cons: non-response

adjustment to cultural differences, special needs

Survey types

Opposite for interview-administered surveys

Survey questions

Objective versus subjective questions:Objective - there is a verifiably true answer

Subjective - based on perspective of respondent

Open versus closed questions:Open - empty space for answer

Closed - pre-set response format options

Response formats

Dichotomous and Multichotomous

Multiple response

Verbal frequency scale (Never ... Often)

Ranking (in order Ordinal)

Likert scale (equal distances)

Graphical rating scale (e.g., line)

Semantic differential (opposing words)

Non-verbal (idiographic)

Level of measurement

Categorical/NominalArbitrary numerical labels

Could be in any order

OrdinalOrdered numerical labels

Intervals may not be equal

IntervalOrdered numerical labels

Equal intervals

RatioData are continuous

Meaningful 0

Sampling

Key terms(Target) population

Sampling frame

Sample

SamplingProbabilitySimple (random)

Systematic

Stratified

Non-probability

Convenience

Purposive

Snowball

Non-sampling biases

Acquiescence

Order effects

Fatigue effects

Demand characteristics

Hawthorne effect

Self-serving bias

Social desirability

Descriptives & graphing
(Lecture 3)

Getting to know data

Play with the data

Don't be afraid you can't break data

Check and screen the data

Explore the data

Get intimate with data

Describe the main features

Test hypotheses

Summary: LOM & statistics

If a normal distribution can be assumed, use parametric statistics (more powerful)

If not, use non-parametric statistics (less power, but less sensitive to violations of assumptions)

Descriptive statistics

What is the central tendency?Frequencies, Percentages (Non-para)

Mode, Median, Mean (Para)

What is the variability?Min, Max, Range, Quartiles (Non-para)

Standard Deviation, Variance (Para)

Normal distribution

MeanSD-ve Skew+ve SkewKurtRule of thumbSkewness and kurtosis in the range of -1 to +1 can be treated as approx. normal

Image source: Unknown

Skewness & central tendency

+vely skewed
mode < median < meanSymmetrical (normal)
mean = median = mode-vely skewed
mean < median < mode

Image source: Unknown

Principles of graphing

Clear purpose

Maximise clarity

Minimise clutter

Allow visual comparison

Univariate graphs

Bar graph

Pie chart

Histogram

Stem & leaf plot

Data plot /
Error bar

Box plot

Non-parametric
i.e., nominal or ordinal

}

}

Parametric
i.e., normally distributed interval or ratio

Non-normal parametric data can be recoded and treated as nominal or ordinal data.

Correlation
(Lecture 4)

Covariation

The world is made of covariations.

Covariations are the building blocks of more complex multivariate relationships.

Correlation is a standardised measure of the covariance (extent to which two phenomenon co-relate).

Correlation does not prove causation - may be opposite causality, bi-directional, or due to other variables.

Purpose of correlation

The underlying purpose of correlation is to help address the question:

What is therelationship or

association or

shared variance or

co-relation

between two variables?

What is correlation?

Standardised covariance

Ranges between -1 and +1, with more extreme values indicating stronger relationships

Correlation does not prove causation may be opposite causality, bi-directional, or due to other variables.

Types of correlation

Nominal by nominal:
Phi () / Cramers V, Chi-squared

Ordinal by ordinal:
Spearmans rank / Kendalls Tau b

Dichotomous by interval/ratio:
Point bi-serial rpb

Interval/ratio by interval/ratio:
Product-moment or Pearsons r

Correlation steps

Choose correlation and graph type based on levels of measurement.

Check graphs (e.g., scatterplot):

Linear or non-linear?

Outliers?

Homoscedasticity?

Range restriction?

Sub-samples to consider?

Correlation steps

Consider

Effect size (e.g., , Cramer's V, r, r2)

Direction

Inferential test (p)

Interpret/Discuss

Relate back to hypothesis

Size, direction, significance

Limitations e.g.,Heterogeneity (sub-samples)

Range restriction

Causality?

Interpreting correlation

Coefficient of determinationCorrelation squared

Indicates % of shared variance

Strength r r2 Weak: .1 - .3 1 10% Moderate: .3 - .5 10 - 25% Strong: > .5 > 25%

Assumptions & limitations

Levels of measurement

Normality

LinearityEffects of outliers

Non-linearity

Homoscedasticity

No range restriction

Homogenous samples

Correlation is not causation

Exploratory factor analysis
(Lecture 5)

What is factor analysis?

Factor analysis is a family of multivariate correlational data analysis methods for summarising clusters of covariance.

FA summarises correlations amongst items into a smaller number of underlying fuzzy constructs (called factors).

Steps / process

Test assumptions

Select extraction method and rotation

Determine no. of factors
(Eigen Values, Scree plot, % variance explained)

Select items
(check factor loadings to identify which items belong in which factor; drop items one by one; repeat)

Name and define factors

Examine correlations amongst factors

Analyse internal reliability

Compute composite scores

Lecture
6

See also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorialSee also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial

Assumptions

Sample size 5+ cases per variables
(ideally 20+ cases per variable)

Another guideline: N > 200

Bivariate & multivariate outliers

Factorability of correlation matrix
(Measures of Sampling Adequacy)

Normality enhances the solution

Types of FA

PAF (Principal Axis Factoring):
Best for theoretical data exploration uses shared variance

PC (Principal Components):
Best for data reduction uses all variance

Rotation

Orthogonal (Varimax) perpendicular (uncorrelated) factors

Oblique (Oblimin) angled (correlated) factors

Consider trying both ways Are solutions different? Why?

How many factors to extract?Inspect EVs look for > 1 or sudden drop (inspect scree plot)

% of variance explained aim for 50 to 75%

Interpretability does each factor 'make sense'?

Theory does the model fit with theory?

Factor extraction

Item selection

An EFA of a good measurement instrument ideally has:a simple factor structure (each variable loads strongly (> +.50) on only one factor)

each factor has multiple loading variables (more loadings greater reliability)

target factor loadings are high (> .6) and cross-loadings are low (< .3), with few intermediate values (.3 to .6).

Psychometrics
instrument
development
(Lecture 6)

Psychometrics

Science of psychological measurement

Goal: Validly measure individual psychosocial differences

Develop and test psychological measures e.g., usingFactor analysis

Reliability and validity

Concepts & their measurement

Concepts name common elements

Hypotheses identify relations between concepts

Brainstorm indicators of a concept

Define the concept

Draft measurement items

Pre-test and pilot test

Examine psychometric properties

Redraft/refine and re-test

Measurement error

Deviation of measure from true score

Sources:Non-sampling (e.g., paradigm, respondent bias, researcher bias)

Sampling (e.g., non-representativeness)

How to minimise:Well-designed measures

Reduce demand effects

Representative sampling

Maximise response rate

Ensure administrative accuracy

Reliability

Consistency or reproducibility

TypesInternal consistency

Test-retest reliability

Rule of thumb> .6 OK

> .8 Very good

Internal consistencySplit-half

Odd-even

Cronbach's alpha

Validity

Extent to which a measure measures what it is intended to measure

Multifaceted

Compare with theory and expert opinion

Correlations with similar and dissimilar measures

Predicts future

Composite scores

Ways of creating composite (factor) scores: Unit weightingTotal of items or

Average of items
(recommended for lab report)

Regression weighting Each item is weighted by its importance to measuring the underlying factor (based on regression weights)

Writing up
instrument development

Introduction Review constructs & previous structures

Generate research question

Method Explain measures and their development

Results Factor analysis

Reliability of factors

Descriptive statistics for composite scores

Correlations between factors

Discussion Theory? / Measure? / Recommendations?

Multiple linear regression
(Lectures 7 & 8)

General steps

Develop model and hypotheses

Check assumptions

Choose type

Interpret output

Develop a regression equation
(if needed)

Linear regression

Best-fitting straight line for a scatterplot of two variables

Y = bX + a + ePredictor (X; IV)

Outcome (Y; DV)

Least squares criterion

Residuals are the vertical distance between actual and predicted values

MLR assumptions

Level of measurement

Sample size

Normality

Linearity

Homoscedasticity

Collinearity

Multivariate outliers

Residuals should be normally distributed

These residual slides are based on Francis (2007) MLR (Section 5.1.4) Practical Issues & Assumptions, pp. 126-127 and Allen and Bennett (2008)

Note that according to Francis, residual analysis can test:Additivity (i.e., no interactions b/w Ivs) (but this has been left out for the sake of simplicity)

Level of measurement and dummy coding

Levels of measurementDV = Continuous (Likert or ratio + normal)

IV = Continuous or dichotomous

Dummy codingConvert complex variable into series of dichotomous IVs

General steps

Develop model and hypotheses

Check assumptions

Choose type

Interpret output

Develop a regression equation
(if needed)

Multiple linear regression

Multiple IVs to predict a single DV:

Y = b1x1 + b2x2 +.....+ bixi + a + eOverall fit: R, R2, and Adjusted R2

CoefficientsRelation between each IV and the DV, adjusted for the other IVs

B, , t, p, and sr2

TypesStandard

Hierarchical

Stepwise / Forward / Backward

Summary:
Semi-partial correlation (sr)

In MLR, sr is labelled part in the regression coefficients table SPSS output

Square these values to obtain sr2, the unique % of DV variance explained by each IV

Discuss the extent to which the explained variance in the DV is due to unique or shared contributions of the IVs

Residual analysis

Residuals are the difference between predicted and observed Y values

MLR assumption is that residuals are normally distributed.

Examining residuals also helps assess: Linearity

Homoscedasticity

Interactions

In MLR, IVs may interact to: Increase the IVs' effect on the DV

Decrease the IVs' effect on the DV

Model interactions using hierarchical MLR: Step 1: Enter IVs

Step 2: Enter cross-product of IVs

Examine change in R2

Analysis of change

Analysis of changes over time can be assessed by: Standard regression Calculate difference scores
(Post-score minus Pre-score) and use as a DV

Hierarchical MLR Step 1: Partial out baseline scores

Step 2: Enter other IVs to help predict variance in changes over time.

Writing up an MLR

Introduction Establish purpose

Describe model and hypotheses

Results Univariate descriptive statistics

Correlations

Type of MLR and assumptions

Regression coefficients

Discussion Summarise and interpret, with limitations

Implications and recommendations

Power & effect size
(Lecture 9)

Significance testing

Logic At what point do you reject H0?

History Started in 1920s & became very popular through 2nd half of 20th century

Criticisms Binary, dependent on N, ES, and critical

Practical significance Is an effect noticeable?

Is it valued?

How does it compare with benchmarks?

Inferential decision making

Statistical power

Power = probability of detecting a real effect as statistically significant

Increase by:

N

critical

ES

Power > .8 desirable

~ .6 is more typical

Can be calculated prospectively and retrospectively

Effect size

Standardised size of difference or strength of relationship

Inferential tests should be accompanied by ESs and CIs

Common bivariate ESs include: Cohens d

Correlation r

Cohens d - not in SPSS - use an online effect size calculator

Confidence interval

Gives range of certainty

Can be used for B, M, ES etc.

Can be examined Statistically (upper and lower limits)

Graphically (e.g., error-bar graphs)

Publication bias

Tendency for statistically significant studies to be published over non-significant studies

Indicated by gap in funnel plot file-drawer effect

Counteracting biases in scientific publishing:

low-power studies tend to underestimate real effects

bias towards publish sig. effects over non-sig. effects

Academic integrity

Violations of academic integrity are evident and prevalent amongst those with incentives to do so: Students

Researchers

Commercial sponsors

Adopt a balanced, critical approach, striving for objectivity and academic integrity

Unit outcomes

Learning outcomes

Design and conduct survey-based research in psychology;

Use SPSS to conduct and interpret data analysis using correlation-based statistics, including reliability, factor analysis and multiple regression analysis;

Communicate in writing the results of survey-based psychological research

Graduate attributes

Display initiative and drive, and use organisation skills to plan and manage workload

Employ up-to-date and relevant knowledge and skills

Take pride in professional and personal integrity

Use creativity, critical thinking, analysis and research skills to solve theoretical and real-world problems

Feedback

What worked well?

What could be improved?

Direct feedback
(e.g., email, discussion forum)

Interface Student Experience Questionnaire (ISEQ).

Open Office Impress

This presentation was made using Open Office Impress.

Free and open source software.

http://www.openoffice.org/product/impress.html

Column 1Column 2Column 3

Row 19.13.24.54

Row 22.48.89.65

Row 33.11.53.7

Row 44.39.026.2

Click to edit the title text format

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline LevelNinth Outline Level







summary and conclusion - survey research and design in psychology

Education