1

Bias

Introduction of issue and background papers

Sebastian Schneeweiss, MD, ScD

Professor of Medicine and Epidemiology

Division of Pharmacoepidemiology and Pharmacoeconomics, Dept of Medicine, Brigham & Women’s Hospital/ Harvard Medical School

Potential conflicts of interest PI, Brigham & Women’s Hospital DEcIDE Center for

Comparative Effectiveness Research (AHRQ) PI, DEcIDE Methods Center (AHRQ) Co-Chair, Methods Core of the Mini Sentinel System (FDA) Member, national PCORI Methods Committee No paid consulting or speaker fees from pharmaceutical

manufacturers Consulting in past year:

WHISCON LLC, Booz&Co, Aetion

Investigator-initiated research grants to the Brigham from Pfizer, Novartis, Boehringer-Ingelheim

Multiple grants from NIH

2

Objective of Comparative Effectiveness Research

3

Efficacy (Can it work?)

Effectiveness* (Does it work in routine care?)

* Cochrane A. Nuffield Provincial Trust, 1972

Placebo comparison (or usual care)

Active comparison (head-to-head)

Most RCTs for drug approval

Goal of CER

Effectiveness = Efficacy X Adherence X Subgroup effects (+/-)

Reality of routine care RCT

4

Baseline randomization

Non-randomized

Primary data

Secondary data

CER

Primary data

Secondary data

Challenges of observational research

Measurement /surveillance-related biases Informative missingness/ misclassification

Selection-related biases Confounding Informative treatment changes/discontinuations

Time-related biases Immortal time bias Temporality Effect window

(Multiple comparisons) 5

6 H-C C and CC, BMJ 2010

Informative missingness: Unintended effects of statins

Data source: QResearch EMR system, England & Wales

7

Confounding (Obs) and informative censoring (Obs and RCT) By indication By contraindicaton, healthy users Adherence

Severity, comorbidity, prognosis

Side effect, treatment

failure

Treat A

Treat B

Patient follow-up

Rx

Rx Rx Rx

Patients Rx

Rx

8

Immortal time bias in an event-based cohort

Cohort entry is defined by a new diagnosis “>” but time until 1st drug exposure is misclassified

Cohort entry is defined by - 1st drug use for exposed - an event (Dx) for the unexposed

Suissa S, AJE 2008

Opportunities: Confounding

Utilize naturally occurring variation in the healthcare system (Dylan Small) Between providers Between systems Between regions Between time periods

Measure naturally occurring variation and capture via propensity score analyses

Negative controls (Prasad & Jena)

9

Opportunities: Censoring

As treated vs. intention to treat analyses (Miguel Hernan)

Inverse probability of discontinuing weighting And similar methods that rely on characterizing the factors

leading to treatment change

10

Opportunities: Transparency about choices

No single study(design) will satisfy the information needs of a decision maker

Need to understand the desired study characteristics and transparent choose accordingly: Internal validity External validity Precision Timeliness Logistical constraints etc

11

4

Intrinsic Study Characteristics Internal validity (bias) External validity (generalizability, transportability) Precision Heterogeneity in risk or benefit (personalized evidence) Ethical consideration (equipoise) External Study Characteristics Timeliness (rapidly changing technology, policy needs) Logistical constraints (study size, complexity, cost) Data availability, quality, completeness

13

Helpful references include:

Rothwell PM Lancet 2005

Miler FG & Joffe S NEJM 2001

Concato J PDS 2012

YES-Prefer baseline randomization for:

• high validity in the presence of strong baseline confounding

• if no ethical issues prevent randomization

• if sufficient resources available

• if enough time available to await results

NO-Prefer observational study for:

• high representativeness for “routine care” by not perturbing the care system

• Need good reason to believe that confounding can be controlled through adjustment

Is baseline randomization indicated?

Below are general considerations that may be neither comprehensive nor applicable for every scenario. These considerations are meant to be amended and changed as more experience is gained with this tool.

Design

14

Intrinsic Study Characteristics Internal validity (bias) External validity (generalizability, transportability) Precision Heterogeneity in risk or benefit (personalized evidence) Ethical consideration (equipoise) External Study Characteristics Timeliness (rapidly changing technology, policy needs) Logistical constraints (study size, complexity, cost) Data availability, quality, completeness

Opportunities: Study portfolio

Often we need multiple studies with different data sources (prim/sec) and different designs

Is there an optimal way to arrange multiple studies so that they complement each other (speed, validity, generalizability) and collectively provide most valid and comprehensive

information for decision makers?

15

Opportunities: Investigator error

Training Guidance ‘Standards’

16

17

Subgroup Analysis ?

Basic Design Consideration

Subgroup definition

Prior pharmacology knowledge

Prior clinical Knowledge

Yes

Cohort study (case-control, case-cohort sampling)

Exposure/outcome considerations

Exposure definition Outcome Definition

Comparison group considerations Clinical meaningfulness

Incident user design considerations

Exposure risk window considerations Case validation necessary?

Specificity and sensitivity of measurement

Yes Consider case-crossover design

no

Meaningful exposure variation within patients?

Defining covariates based on clinical knowledge

Defining additional covariates empirically (high-dimensional proxy adjustment)

Demonstrate covariate distributions by treatment group with RDs and 95% CIs

Supplemental covariate information required that is not available in primary data source?

Collect additional information in subpop. • 2-stage sampling • External data source

-(PS Calibration) - Multiple imputation

Yes

Propensity score (PS) analysis Missing covariate values in EMRs? Multiple imputation

Estimating propensity score

Yes

Explore effect measure modification by PS: tabulate RR,

RD for each PS stratum

Graphically explore PS distribution by treatment group

Yes Effect measure modification by PS?

•Stratify by PS deciles •Match on PS (1:1, 1:n, 1:n:m)

Trim 5% of patients on each end of PS distribution or match by PS

Balancing Patient Characteristics

Demonstrate covariate balance by treatment group with RDs and 95% CIs

19

Repeat analyses after changes in: • Definition of “incident users” • Definition of exposure risk window • Outcome definition if appropriate

Explore changes in effect estimates after making structural assumptions about unmeasured confounders

Sensitivity Analyses

Statistical analysis*

*For illustration purposes only an analysis after PS matching is shown.

Calculate risk difference (RD) and risk ratio (RR);

95% confidence intervals (CIs) for main result. Report person-time (p-t), number of events

Subgroup analysis Calculate RR, RD for

each subgroup

Dose-response analysis

Include time since initiation as subgroup

Report

Opportunities: Transparency

Sharing of data vs. sharing of analytics environment Data stay were they are but analytics infrastructure allows

other investigators (in collaboration) to re-analyze data

20