comparing research designs fw 2013 handout version
Post on 08-May-2015
1.310 Views
Preview:
DESCRIPTION
TRANSCRIPT
Comparing Research Designs
Patrick BarlowStatistical and Research Design Consultant, Graduate School
of Medicine, UTKPhD Candidate in Evaluation, Statistics, and Measurement,
UTK
On the Agenda
Important considerations in research design Reliability & validity Biases & confounding Strength of evidence
Observational Research Designs Cross-sectional study Case-control study Cohort study
Experimental Research Designs The Basics of Factorial and Crossover Trials
Important considerations
in research designConfoundin
gBias
ReliabilityValidity
Reliability & Validity
Reliability Validity Refers to the consistency of
an instrument/measurement.
Thought of as an individual’s “true score” on the phenomenon you aim to measure minus “measurement error”
Two common types of reliability Internal consistency:
Cronbach’s alpha, KR20 Inter-Rater: Kappa
statistic
Necessary but not sufficient in determining validity.
Refers to the accuracy of an instrument/measurement
In other words, “the degree to which you’re measuring what you claim to measure”
Two broad types of validity Internal validity External validity
Internal vs. External Validity
One of the strengths of randomized designs are that they have substantially higher internal & external validity.
Internal Validity: refers to the integrity of the experiment itself. It is the ability to draw a causal link between your treatment and the dependent variable of interest.
External Validity: refers to the ability to generalize your study findings to the population at large. In other words, are your findings from a sample of UTMCK patients with HTN going to apply to all patients with HTN?
Threats to Internal Validity
Concerns the accuracy of measurement within the study
Shadish, Cook & Campbell (2002) summarized a number of possible threats to internal validity, which can severely jeopardize the findings of a study. In particular: History, Mortality, & Maturation Repeated Testing Confounding Diffusion & Compensatory Rivalry
Threats to Internal Validity
Diffusion & Compensatory Rivalry Diffusion: Treatment effects can “spill over” or “spread” across
treatment groups. EX: Patients from different groups live near each other and discuss / share their experiences or treatments.
Compensatory Rivalry: Patients perform in a certain way because they know they’re in the control / experimental groups.
Threats to Internal Validity
History, Mortality, & Maturation History: events external to the experiment influence the
participants’. EX: Superstorm Sandy hits during a crossover trial in New Jersey.
Mortality: Patients either die (mortality) or drop out of the study (attrition) at different rates.
Maturation: Patients change over the course of the treatment, which influences results. EX: Children grow up during the course of a pediatric clinical trial.
Repeated Testing Patients can become “test-wise” if given the same subjective
test multiple times, or they become conditioned to being tested (EX: patient’s pulse increases before a needle stick).
External Validity
The ability to generalize the findings of your study to the relevant population.
Threatened by Bias Confounding Non-experimental design (i.e. case-control vs. RCT) Lack of randomization
External validity is the strongest when a true experimental design is used.
Confounding
A confounder is a variable that is causally associated with the outcome (DV) and may or may not be causally associated with the exposure (IV)
Causes spurious conclusions & inferences to be made about a set of variables
Reduced through Randomization Matching Statistically controlling (covariates)
Confounding Example
Smoking Hx
HPV
Cervical Cancer
?
Bias in Research
The result of systematic error in the design or conduct of a study
Can artificially “trend” results Toward the Null hypothesis Toward the Alternative
hypothesis
A major problem to consider when planning any study
Common Biases
Selection bias: one relevant group in the population (e.g. cases positive for predictor variable) has a higher probability of being included in the sampleMisclassification can be either unsystematic
(random) or systematic (bad)
Information: bias from erroneously classifying people in exposure/outcome categoriesRecall/Response: bias associated with
inaccurate recall of exposure or representation of true exposure (self-report)
Experimenter/Interviewer bias: Differential treatment of participants in treatment and control groups
Publication: the tendency to publish only “positive” or “significant” findings.
Strength of EvidenceThe Bradford Hill Criteria
Provides researchers with seven criteria for assessing strength of evidence.
Strength of association (i.e. effect size) Consistency (i.e. reliability) Specificity Temporal relationship Biological gradient Plausibility Coherence Experiment (reversibility) Analogy (consideration of alternate explanations)
Pyramid of Clinical Evidence
RCTCohort Studies
Case Control Studies
Case Series
Case Reports
Ideas, Editorials, Opinions
Animal research
In vitro (‘test tube’) research
Systematic Reviews & Meta-
analyses
Evidence Summaries
Level 2 Evidence
Level 1 Evidence
Level 3 Evidence
Cross-Sectional Studies: Level
2.3
Observational Research Designs
Cross-sectional
Case-controlCohort
Cross-Sectional Studies
“Snapshot” of a population.
People are studied at a “point” in time, without follow-up.
Strength of evidence…
What are some research questions that can be answered with cross-sectional designs?
Advantages and Disadvantages of Cross-
Sectional StudiesAdvantages Disadvantages Fast and inexpensive No loss to follow-up Springboard to
expand/inform research question
Can target a larger sample size
Can’t determine causal relationship
Impractical for rare diseases
“Garbage in, garbage out”
Risk for nonresponse
Case-Control Studies
Always retrospective Prevalence vs. Incidence
A sample with the disease from a population is selected (cases).
A sample without the disease from a population is selected (controls).
Groups are compared using possible predictors of the disease state.
Advantages and Disadvantages of Case-Control
StudiesAdvantages Disadvantages
High information yield with few participants
Useful for rare outcomes
Cannot estimate incidence of disease
Limited outcomes can be studied
Highly susceptible to biases
Strategies for Sampling Controls
Population versus hospital/clinic-based controls
Matching Individual level Group level
Using two or more control groups
Cohort Studies
A “cohort” is a group of individuals who are followed or traced over a period of time.
A cohort study analyzes an exposure/disease relationship within the entire cohort.
Groups selected based on exposure to a risk factor.
Level of evidence?
Cohort Design
Prospective vs. Retrospective Cohort
StudiesExposure Outcome
Prospective
Assessed at the beginning of the study (present)
Followed into the future for outcome
Retrospective
Assessed at some point in the past
Outcome has already occurred
Advantages and Disadvantages of Cohort
StudiesAdvantages Disadvantages
Establish population-based
incidence
Temporal relationship inferred
Time-to-event analysis
possible
Used when randomization not
possible
Reduces biases (selection,
information)
Lengthy and costly
Not suitable for rare/long-
latency diseases
May require very large
samples
Nonresponse, migration and
loss-to-follow-up
Sampling, ascertainment and
observer biases
Experimental Designs
The Basics of Factorial and Cross-Over Designs
Experimental DesignsWhat are They?
Considered to be the “gold standard” of clinical
evidence because:
Randomization is used to reduce the effect of biases
and confounding variables
Patients (single) and researchers (double) can be
blinded to the intervention
High internal and external validity allow for assessing
cause and effect relationships.
The most basic experimental design is a
“Parallel trial.”
Patients are randomized into one of two groups, and
remain in the same group throughout the study.
“Double-blind trials”
Factorial DesignsWhat are They?
Factorial designs allow for researchers to test multiple interventions or treatment combinations in a single study. For example: drug A or Drug B and 3x per
week or everyday dose cycle.
The simplest form of this design is a 2x2 factorial design.
Allows researchers to test individual treatment effects and/or interactions between different treatments.
Looks like a “grid”
Factorial DesignsWhy are They Used?
Factorial design are commonly used to effectively test multiple treatments or “Main effects” in a single study. More efficient and more statistically powerful than multiple single
intervention studies
Especially useful for testing interactions among different interventions or treatments
Main Effects
Interactions
Factorial DesignsExample
Dose Cycle
StatinRosuvastat
in (Crestor)
Atorvastatin (Lipitor)
3x Per Week
M LDL M LDL
Everyday M LDL M LDL
What is the effect of dose (3x pw or everyday) and statin (Rusuvastatin or Atorvastatin) regimen on mean LDL Cholesterol?
Cross-over DesignsWhat are They?
A cross-over trial design involves giving the two or more interventions/treatments to a single group of patients.
At its most basic, this trial tests the efficacy of two treatments where each patient spends a period of time under both treatment options.
Patients are randomized into which treatment they receive first, and then swap to the other treatment after a predetermined time.
Cross-over DesignsWhat are They?
A
B“Cross-over”
A
B
Cross-Over DesignsWhy are They Used?
Cross-over trials are useful because they reduce confounding factors associated with between-subjects designs. Patients serve as their own controls Useful for time-dependent research questions Higher statistical power than between subjects designs due to no
between-subjects error (i.e. need less patients to find statistical significance).
Cross-Over DesignsExample
3x Per Week
Treatment
Everyday Treatmen
t
Everyday Treatme
nt
3x Per Week
Treatment
Week Six
Disadvantages of RCT Designs
Extremely time and resource demanding
Unethical in many situations
Poor external validity if the RCT is too highly controlled
Difficult to study rare events
Therapeutic misconception
In Pairs…
Work together to brainstorm an example of how your topic could be addressed using 1) a Cross-Sectional design, 2) a case-control design, 3) a prospective or retrospective cohort design, and an RCT (Parallel, factorial, or cross-over).
Be prepared to share your responses
top related