review of "survey research methods & design in psychology"
TRANSCRIPT
Survey Design II
Lecture 12
Survey Research & Design in Psychology
James Neill, 2011
Survey Research & Design in Psychology:Review & Summary
7126/6667 Survey Research & Design in PsychologySemester 1, 2011, University of Canberra, ACT, AustraliaJames T. NeillHome page: http://ucspace.canberra.edu.au/display/7126Lecture page: http://ucspace.canberra.edu.au/display/7126/Lecture+-+ReviewNotes page:http://en.wikiversity.org/wiki/Survey_research_methods_and_design_in_psychology
Image name: Recycle symbol Taiwan.svgImage source: http://commons.wikimedia.org/wiki/File:Recycle_symbol_Taiwan.svgImage author:Bryan Derksen http://commons.wikimedia.org/wiki/User:Bryan_DerksenLicense: Public domain
Description:Reviews this semester-long (150-hour), third year undergraduate psychology research unit which focused on survey research methods and survey design. This lecture emphasises the second-half of the unit's content on MLR, ANOVA, significance testing, power, and effect size, as well as providing advice about the lab report and final exam assessment items.
Overview
Review
Unit aims and outcomes
Research process
Survey design
MLR, ANOVA, ES & Power
What type of analysis? (decision tree)
AssessmentLab report
Final exam
Evaluation & feedback
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domain
Image source: http://commons.wikimedia.org/wiki/File:Autoroute_icone.svgLicense: CC-BY-A 2.5Author: http://commons.wikimedia.org/wiki/User:Doodledoo
Unit aims & outcomes
http://en.wikiversity.org/wiki/Survey_research_and_design_in_psychology/Overview
Aims & outcomes
Knowledge & skills
for conducting ethical,
well-designed, survey-based research in psychology.
How confident are you that could conduct a good quality survey-based research study?
For 4th year
Honours?
In the
work-place?
Aims & outcomes
Theory & practice
of survey-based research, incl.:
Research questions / hypotheses
Survey design
Sampling
Interpreting & communicating results
Aims & outcomes
Use of SPSS for:
Data entry
Correlations
Factor analysis
Qualitative analysis
Reliability
MLR
Advanced ANOVA
Research process
Research process
1. Establish
need for info/
research2. Problem
definition/
Hypotheses3. Researchdesign4. Sampling/Data collection5. Data
analysis6. Reporting
Image source: James Neill, Creative Commons Attribution-Share Alike 2.5 Australia, http://creativecommons.org/licenses/by-sa/2.5/au/
Funnel model
Image source: Unknwn
Survey design
Survey design
Operationalisation of fuzzy concepts
Question types (objective/subjective, open/closed)
Response formats (e.g., dichotomous, frequency, Likert, multiple response, idiographic)
Levels of measurement (LOM)
Reliability & validity
Sampling (probability/non-probability)
Modes of administration (self-report (f2f, mail, web), interview (f2f/phone))
Items should
measure different
aspects of thelatent (underlying)
constructLatent Construct
Measured
Construct
Image source: James Neill, Creative Commons Attribution-Share Alike 2.5 Australia, http://creativecommons.org/licenses/by-sa/2.5/au/
Latent Construct
Measured ConstructPoor items
will create
brown sludge
(noisy
(unreliable)
measure)
Image source: James Neill, Creative Commons Attribution-Share Alike 2.5 Australia, http://creativecommons.org/licenses/by-sa/2.5/au/
Describing data
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domain
Describing data
Data screening (out of range, invalid etc.)
Discrete data: Frequencies & %s
Continuous data:
4 moments of a normal distributionCentral tendency
Dispersion
Skewness
Kurtosis
Visualisation of data
Aids interpretation of descriptives and tests of differences or relationships.
Univariate: histogram, bar graph, error-bar graph
Bivariate: scatterplot, clustered bar graph
Multivariate: Venn diagrams, multiple line graph, 3-d scatterplot
Software for
data visualisation (graphing)
Statistical packages
e.g., SPSS Graphs or via Analyses
Spreadsheet packages
e.g., MS Excel
Word-processors
e.g., MS Word Insert Object Micrograph Graph Chart
Correlations
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domain
Correlations
Strength & direction of bivariate linear relations
Building block for FA & MLR
Non-parametric correlations
(e.g., Point bi-serial, Phi/Cramer's V)
Scatterplots watch out for: Outliers
Non-linearity
Limited range
Caution with causal interpretation
Factor analysis
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domain
Factor analysis
Purpose Data reduction
Developing reliable & valid measures of fuzzy construct
Assumptions Linear relations
Sample size
(min. 5 cases: 1 variable)
Factor analysis
ExtractionPC vs. PAF
Rotation methodVarimax vs. Oblimin
Number of factorsKaisers criterion
Scree plot
Theoretical structure
Interpretation of factor loadings
Factor analysis
Decide how many factors
Iteratively eliminate itemsCommunality > .5?
Primary loadings > .5?
Cross-loadings < .3?
Diff. btw primary & cross-loadings > .2?
Sufficient items per factor
Face validity
Correlations between factors
Compare model across groups
Compare models across groups
% variance explained
No. of factors
Item loadings
Reliabilities & composite scores
Internal reliability: For each factor, calculate Cronbach s : > .8 = very good
> .7 = good
> .6 = OK
Composite scoresUnit-weighting
Regression-weighting
Reversing a scale e.g.,IM = mean(item1,item2,item3)EM = mean (item4,item5,item6)M = IM + (8 EM)1 2 3 4 5 6 77 6 5 4 3 2 1
Qualitative analysis
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domain
Qualitative analysis
Purposes Pilot study, exploratory research
Theory testing; Data reduction (synthesise meaning)
Validity-testing
Methods Quantitative (Content analysis, Multiple response analysis, Graph (e.g., bar graph)
Qualitative (Thematic analysis)
Inferential statistical
decision making tree
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domain
Statistical decision tree
Establish a research question and/or hypothesisDifferences or relationships?
No. of IVs and DVs
Identify levels of measurement
Use a statistical decision tree to identify an appropriate analysis
SeeInferential statistics decision-making tree
Multiple linear regression
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domain
Multiple linear regression
Linear regression formula
Y hat = ax + bY = ax + b + eProportion of variance in a DV explained by one or more IVsR, R2, Adjusted R2
F (significance of R)
Change in R2 (Hierarchical)
Multiple linear regression
Assumptions: LOMContinuous DV
Dichotomous or continuous IVs
Normality (helps satisfy other assumptions)
Linearity
Homoscedasticity
Multicollinearity
Multivariate outliers
Multiple linear regression
Methods Standard / Direct
Hierarchical
Forward
Backward
Stepwise
Overall hypothesis: (Null) That the IVs do not explain variance in the DV (i.e., that R is 0)
+ one hypothesis per predictor: (Null) That each IV is not a significant predictor of variance in the DV (i.e., that t for each predictor is less than the critical value)
Multiple linear regression
Consider/interpret:Direction of each predictor
Size of each predictor ( - standardised)
When IVs are correlated, interpret zero-order vs. partial correlations
sr2 = % variance in DV explained by each IV
Can use Venn or path diagrams to depict relationships between variables
Multiple linear regression
ANOVA
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domain
ANOVA
Extension of t-test
Like MLR, ANOVA involves:One continuous DV (although ANOVA can handle multiple DVs)
One or more IVs
Unlike MLR, in ANOVA:Interactions are automatically tested
IVs must be categorical
Significant results may indicate need for followup or post-hoc tests
Types of ANOVA
1-way ANOVA
1-way repeated measures ANOVA
2-way factorial ANOVA
Mixed design ANOVA
(Split-plot ANOVA)
ANCOVA
MANOVA
ANOVA
AssumptionsCell size > 20 (Ideal)
Normally distributed DVs (for each level of the IVs)
Homogeneity of variance (b/w subjects)
Sphericity (w/in subjects)
Follow-up tests:Post-hoc tests
Planned comparisons
Effect sizes: 2 , p2 and Cohens d
Power, effect size, significance testing, publication bias & academic integrity
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domain
Significance testing
Significance testing has dominated psychology, but is problematic, mainly because:Results are dichotomous (sig. or not), which doesn't help us to understand the size of effect.
Sig. test results are influenced by power esp. if particularly high or low.
Power & effect sizes
Power and effect sizes have been neglected. Therefore:Calculate the power of studies (prospectively &/or retrospectively)
Report ESs & CIs to complement inferential statistics
r, r2, R, R2
2, p2, d
Publication bias &
academic integrity
Publication bias (low power; favouritism of sig. findings; funnel plots)Academic integrity - Integrity is doing the right thing, especially when no one is watching.
Lab report
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domain
Lab report - Tips
Check the lab report guidelines and the marking criteria
Use example HD lab report and sample write-ups as guides
Demonstrate your knowledge via independent thinking/work
Tell a story
Lab report - Structure
Cover Sheet
Title Page
Abstract
Body
References
Appendices
Lab report - Introduction
Explain the rationale what is the problem to be solved?
No waffle cut to the chase only review literature or argument relevant to the RQs / hypotheses
State clear RQs and/or hypotheses One per test/analysis/effect
Lab report - Method
Well-organised
(like a recipe)
Present relevant details efficiently (avoid getting bogged down in extraneous detail)
Lab report - Method
Sections:
Participants
Materials or Instrumentation
Procedure
Replicable?
A nave reader must be able to replicate the study
Lab report Results
Data screening
Consider LOM assumptions
Caution in use of overall scores
132132Overall Score not validOverall Score validImage source: James Neill, Creative Commons Attribution-Share Alike 2.5 Australia, http://creativecommons.org/licenses/by-sa/2.5/au/
Lab report Results (EFA)
EFA:PC/PAF?
Varimax/Oblimin?
2-6 factors?
5-30 items removed?
50%-60% of variance?
Table of factor loadings and communalities
Correlations between factors
Internal consistency for each factor
Lab report Results (MLR)
Example MLR: Hierarchical:DV = Campus Satisfaction
Step 1IV1 = Gender (M / F)
Step 2IV1 = Planning TM (Continuous)
IV2 = Time wasting TM (Continuous)
R2, Adjusted R2, Change in R2
Table of correlations and regression coefficients
Lab report Results (ANOVA)
ANOVA: e.g., 2 x (3) Mixed ANOVABetween-subjects IV:
Enrolment Status (FT / PT)
Within-subjects DV:
Satisfaction (Educational / Social / Campus)
Table of cell and marginal descriptives (M, SD, Sk, Kurt) + Graph
Effect sizes (p2 , d)
Lab report Results (Qualitative)
Example Qualitative analysis: Least or most satisfactory aspects of UCResearch question?
Qualitative or quantitative approach?
Table of main themes (names, description, example quotes, frequency/%)
Bar graph?
Thick description?
Sub-samples?
Lab report - Discussion
Provide insight about the results
Draw conclusions about the RQs & hypotheses in light of the results.
Discuss key strengths & limitations of the study. (Balanced criticism)
Draw out implications and recommendations
Lab report - Discussion
Offer specific, practical recommendations e.g.,
Theory: What are the implications for the theory/rationale upon which the study was based?
Methods: How could the research design (e.g., instrumentation) be improved?
Practice: Implications for students and universities e.g., for improving satisfaction?
Lab report Appendices
Optional: Include appendices where relevant and referred to in the body text. Appendices may not be consulted by a reader, so if its important make sure key content is covered in the text.
Use for content which would break the flow, but which is relevant to understanding the study e.g., the EFA correlation matrix.
Lab report Appendices
APA style not necessary.
Use headings (e.g., Appendix A, B, C etc.) and possibly titles
e.g.,
Appendix A:
Bivariate correlations amongst the university student satisfaction
items
Lab report - Submission
Insert & complete the Cover Sheet
Submit ONE DOCUMENT containing the coversheet, lab report and appendices
NO EXTRA ATTACHMENTS
Upload via the Moodle drop-box
You can re-submit up until the due date (after that, resubmission will incur late penalties)
Lab report - Marking
Lab report marks should be returned before the exam.
Keep an eye on your student email and Moodle Announcements.
Final exam
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domain
Final exam
Computer-based, supervised
Style, structure and content will be similar to the quizzes
~120 multi-choice questions in 180 mins (plenty of time).
Practice exams are available
Final exam
Permitted materials
Non-programmable calculator
Non-annotated foreign language dictionary
Marks and feedback
Upon submitting, you will receive your exam mark and feedback.
Evaluation & feedback
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domain
Evaluation & feedback: USS
Unit Satisfaction SurveyAvailable now on OSIS but you may wish to wait until after assessment results
UC takes these results very seriously ~40% response rate
Possible issues & topics
Please contribute your honest feedback:Lectures?
Tutorials?
Textbooks?
Assessment?
Website(s)?
Software SPSS?
Workload?
You are also welcome to
tell me directly or
email me or
post feedback on the
Moodle discussion forum.
Open Office Impress
This presentation was made using Open Office Impress.
Free and open source software.
http://www.openoffice.org/product/impress.html
Click to edit the title text format
Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline LevelNinth Outline Level
Click to edit the title text format
Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline LevelNinth Outline Level
Click to edit the title text format
Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline LevelNinth Outline Level
Click to edit the title text format
Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline LevelNinth Outline Level