research methods: design and analysis
DESCRIPTION
Research Methods: Design and Analysis. Covering the research cycle, research questions, operationalization of variables, literature review, research designs, sampling method, instrumentation, data collection, validity, reliability, data analysis plan, and sample sizeTRANSCRIPT
Research Methods: Design and Analysis
by Dr. James Lani
Today’s Webinar
To schedule a consultation call 877-437-8622 or email [email protected]
• Why Do Research?• The Research Cycle• Research Questions and (Testable) Hypotheses
• Operationalization of Variables• Literature Review Resources• Research Designs• Sampling Method and Sample Assignment• Instrumentation• Method of Data Collection (Procedures)• Validity
• Internal Validity• External Validity
• Reliability• Data Analysis Plan• Sample Size• Q & A
Why Do Research?
To schedule a consultation call 877-437-8622 or email [email protected]
OK, to get out of school….and also to examine questions we care about. We want to test a theory, evaluate a program or intervention, describe a phenomenon.
Does Violence on TV cause children to be more or less aggressive?Do opposites attract or birds of a feather flock together?Does treatment X predict academic achievement?Is there a difference on depression scores by group (exercise vs. not)?
What is True???Goal of research: describe, explain, and predict. To establish knowledge, you need methods and statistics.
The Research Cycle
To schedule a consultation call 877-437-8622 or email [email protected]
LiteratureReview
MethodHypotheses and
Operational Variables
MethodDesign Study and
Collect Data
ResultsAnalyze data and
accept or reject Ho’s
Discussion
Research Questions and (Testable) Hypotheses
To schedule a consultation call 877-437-8622 or email [email protected]
• Hypotheses: State relationships among variables (constructs). Typically in terms of Differences, Relationships, or Prediction.
• Keep research questions and hypotheses aligned (in number and language).
• Questions must be testable.• Keep it tentative: no proof—just support
or non-support
Operationalization of Variables
(or how do you measure what you’re studying?)
To schedule a consultation call 877-437-8622 or email [email protected]
Constructs—objects that are not directly observable• Constructs are measured with variables• Should be quantifiable • (e.g., depression can be measured with a Beck
depression scale or by the observation of the number of times someone laughs)
Confounding Variables• Provide alternative explanation of response in the DV
Levels of Measurement: Nominal, Ordinal, Interval, and Ratio
Literature ReviewResources
To schedule a consultation call 877-437-8622 or email [email protected]
See: UW—Madison—the writing centerUC Santa Cruz—UNW library—write a literature reviewWashington and Lee University—literature reviewStatistics Solutions’ Free membership—sample dissertation
Research Designs
To schedule a consultation call 877-437-8622 or email [email protected]
EXPLORATORY
EXPERIMENTALDescriptive
Longitudinal
CAUSAL Cross-sectionalCorrelational Cohort
designs
Non-experimental
Sequential
Research Designs
To schedule a consultation call 877-437-8622 or email [email protected]
Its about how you structure your inquiry
Research Designs
To schedule a consultation call 877-437-8622 or email [email protected]
Experimental vs. Non-experimental Designs
Non-Experimental DesignsCorrelational studies, observational
studies, quasi-experimental studies, and surveys
Experimental (or True) DesignsManipulate the IV(e.g., Control vs. Experimental groups)Random assignmentConfounds are controlled or eliminated
Research Designs
To schedule a consultation call 877-437-8622 or email [email protected]
Experimental Designs
One shot case studies X O
One group pretest posttest O1 X O2
Statistic group X O1 O2 R=random assignment, O=observation, X=treatment group
Research Designs
To schedule a consultation call 877-437-8622 or email [email protected]
True Experimental DesignsPretest-posttest control group design
R O1 X O2
R O3 -- O4
R=random assignment, O=observation, X=treatment groupGroup Time 1 Time 2 Time3
Random Treatment Mean X Mean
Assignment Control Mean Mean
No diff Sig diff
Pretest Posttest0
5
10
ControlTreatment
Research Designs
To schedule a consultation call 877-437-8622 or email [email protected]
For great research design references, see:
Cook and Campbell ‘79 Campbell and Stanley ‘63 Creswell ‘05 Leedy and Ormrod ‘10 SocialResearchMethods.net
A word about Sampling Method and Sample Assignment
To schedule a consultation call 877-437-8622 or email [email protected]
Random sample: each individual in the population has an equal chance to become a participant of the sample.
Random assignment: each participant of the sample has an equal chance of being assigned to a group (treatment or control).
Instrumentation
To schedule a consultation call 877-437-8622 or email [email protected]
These are the tools that measure the outcomes—typically variables in the tools are categorized into independent variables (IV’s) and dependent variables (DV’s).
IV’s (or treatment variables) are typically manipulated by researcher.
DV’s (also called outcome variables or criterion variables) are responses to IV, caused by IV, or predicted by IV.
The variables should be reliable and valid.
Don’t make your own.
See Google scholar, review articles, Statistics Solutions directory.
Instrumentation
To schedule a consultation call 877-437-8622 or email [email protected]
Questionnaires Interviews Observations Rating scales Achievement tests Personality
inventories Historical reports or
data
Method of Data Collection (Procedures)
To schedule a consultation call 877-437-8622 or email [email protected]
After describing the sample and the instruments, tell the reader how you are going to systematically and objectively administer these instruments to these participants.
ParticipantsMeasures
Validity
To schedule a consultation call 877-437-8622 or email [email protected]
You measure what you say you are measuring.
(Validity is more important than reliability)
Internal Validity
To schedule a consultation call 877-437-8622 or email [email protected]
Eliminate confounds or competing explanations of differences on DV (E.g., Differences on Self-efficacy by gender…but the woman were 35 y.o. and men were 75 y.o.)
Threats to internal validity• Selection bias—you think they represent the population (Safeguard tip: randomly sample and randomly assign)• Pre-post studies without a control group
o History/maturation—something else occurred between the assessments
• Drop-out/attrition • Reliability of measure• Low power• Order effect• Multiple tests of significance • John Henry Effect
External Validity
To schedule a consultation call 877-437-8622 or email [email protected]
Def: The ability to generalize outcomes to the population
Threats• Population validity—ability to generalize to
population. (E.g., Kohlberg moral development—males value individual rights, where Gilligan found that females valued relationships).
• Ecological validity—ability to generalize from lab to everyday environments.
Reliability
To schedule a consultation call 877-437-8622 or email [email protected]
Inter-rater reliability (kappa coefficient)
Test-retest reliability Internal consistency—
Cronbach’s alpha
Data Analysis Plan(what to include)
To schedule a consultation call 877-437-8622 or email [email protected]
Reliability of scales: Cronbach’s alpha
Descriptive statistics: Means, standard deviations, frequency and percentages.
A data analysis plan for EACH research hypothesis.
• Inferential: ANOVA, MANOVA, regression, time-series, SEM, Chi-square, correlation
• Non-inferential: descriptives (e.g., M, SD, %)
Assumptions of analyses and justification for analyses
Sample Size
To schedule a consultation call 877-437-8622 or email [email protected]
N = ?
1. Effect Size—importance of differences
2. Power—likelihood of finding differences (typically .80)
3. Alpha (typically .05)
We’re interested… Webinar Feedback and
What You’d LikePlease provide feedback on the webinar and tell us what
else you’d like to learn about in upcoming webinars.
Send comments to: [email protected]
Thank You for your Participation
and Attention!
Join us for our next webinar on Wednesday, November 20th at
8:30pm ETResults Chapter:
Conducting, Interpreting, and Writing