a tipping point method to evaluate sensitivity to …...a tipping point method to evaluate...
TRANSCRIPT
![Page 1: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/1.jpg)
A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data AssumptionsCesar TorresSeptember 24, 2019ASA Biopharmaceutical Section Regulatory-Industry Statistics Workshop
![Page 2: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/2.jpg)
2
This presentation reflects information from a manuscript that is a product of joint work with:• Gregory Levin• Daniel Rubin• William Koh• Rebecca Rothwell• Thomas Permutt
![Page 3: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/3.jpg)
3
DISCLAIMERThis presentation reflects the views of the author and should not be construed to represent FDA’s views or policies.
![Page 4: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/4.jpg)
4
OVERVIEW1 Motivation
2 Tipping Point Approach
3 Example
4 Strengths and Limitations
5 Closing Thoughts
www.fda.gov
![Page 5: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/5.jpg)
5
MOTIVATION1
www.fda.gov
![Page 6: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/6.jpg)
6
Missing Data Sensitivity Analyses• Critical to evaluate the sensitivity of conclusions to
violations in missing data assumptions• Sensitivity analyses should:
– Not consist of a few alternative methods/models assuming same missingness mechanism
– Not explore only a local or limited space of violations– Systematically and comprehensively explore the space
of possible assumptionswww.fda.gov
![Page 7: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/7.jpg)
7
Tipping Point Analysis• Independently vary assumptions about missingness
mechanism in each arm• Identify and discuss clinical plausibility of
assumptions (the “tipping points”) under which there is no longer evidence of efficacy
• Typically relies on single or multiple imputation of missing outcomes
www.fda.gov
![Page 8: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/8.jpg)
8
TIPPING POINT APPROACH2
www.fda.gov
![Page 9: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/9.jpg)
9
NotationFor treatment arm ℎ ∈ 𝑡𝑡, 𝑐𝑐 :• 𝜋𝜋ℎ: probability of patient completing study• 𝜇𝜇ℎ: true mean in completers• 𝛿𝛿ℎ: difference in true means between dropouts and
completers– 𝜇𝜇ℎ + 𝛿𝛿ℎ: true mean in dropouts
• 𝑁𝑁ℎ: number of completers• 𝑛𝑛ℎ: sample size
www.fda.gov
![Page 10: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/10.jpg)
10
Parameters of Interest• Mean in treatment arm ℎ is 𝜋𝜋ℎ𝜇𝜇ℎ + 1 − 𝜋𝜋ℎ 𝜇𝜇ℎ + 𝛿𝛿ℎ ≡ 𝜇𝜇ℎ + 1 − 𝜋𝜋ℎ 𝛿𝛿ℎ
• 𝜃𝜃 ≔ 𝜇𝜇𝑡𝑡 + 1 − 𝜋𝜋𝑡𝑡 𝛿𝛿𝑡𝑡 − 𝜇𝜇𝑐𝑐 + 1 − 𝜋𝜋𝑐𝑐 𝛿𝛿𝑐𝑐
www.fda.gov
![Page 11: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/11.jpg)
11
Assumptions• Outcomes of completers are i.i.d. under some distribution with mean 𝜇𝜇ℎ and finite variance 𝜎𝜎ℎ2– Normality is not assumed!
• Outcomes of dropouts have common mean 𝜇𝜇ℎ + 𝛿𝛿ℎ• Completion probability for each patient equal to 𝜋𝜋ℎ ∈ 0, 1• Completion probabilities are independent of realized outcomes• Patients are mutually independent
• 𝑛𝑛𝑡𝑡𝑛𝑛𝑡𝑡+𝑛𝑛𝑐𝑐
→𝑝𝑝𝑟𝑟 ∈ 0, 1
www.fda.gov
![Page 12: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/12.jpg)
12
Proposed Estimator• For assumed values of 𝛿𝛿𝑡𝑡 and 𝛿𝛿𝑐𝑐 , �𝜃𝜃 ≔ �̂�𝜇𝑡𝑡 + 1 − �𝜋𝜋𝑡𝑡 𝛿𝛿𝑡𝑡 − �̂�𝜇𝑐𝑐 + 1 − �𝜋𝜋𝑐𝑐 𝛿𝛿𝑐𝑐
• 𝑉𝑉𝑉𝑉𝑟𝑟 �𝜃𝜃 ∼𝑎𝑎𝑠𝑠𝑡𝑡2
𝑁𝑁𝑡𝑡+ 𝑠𝑠𝑐𝑐2
𝑁𝑁𝑐𝑐+ 𝛿𝛿𝑡𝑡2�𝜋𝜋𝑡𝑡 1−�𝜋𝜋𝑡𝑡
𝑛𝑛𝑡𝑡+ 𝛿𝛿𝑐𝑐2�𝜋𝜋𝑐𝑐 1−�𝜋𝜋𝑐𝑐
𝑛𝑛𝑐𝑐
www.fda.gov
![Page 13: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/13.jpg)
13
EXAMPLE3
www.fda.gov
![Page 14: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/14.jpg)
14
Xeljanz (tofacitinib)• Janus kinase inhibitor previously approved for treatment of
rheumatoid arthritis• Evaluated for safety and effectiveness in psoriatic arthritis• Co-primary endpoints:
– American College of Rheumatology 20% (ACR20) response at Month 3– Change from baseline to Month 3 in Health Assessment Questionnaire-
Disability Index (HAQ-DI) score• More details available in meeting briefing materials for August 3, 2017
Arthritis Advisory Committee Meeting
www.fda.gov
![Page 15: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/15.jpg)
15
HAQ-DI• Patient-reported outcome measure of patient's
level of functional ability• Ranges from 0 to 3, with higher scores being
worse
www.fda.gov
![Page 16: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/16.jpg)
16
Primary Analysis• Mixed effects model for repeated measurements
(MMRM)• Carried out in all randomized patients who received
at least one dose of randomized treatment• Can be viewed as having targeted treatment policy
estimandwww.fda.gov
![Page 17: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/17.jpg)
17
Primary Analysis Results• Statistically significant: p < 0.0001• Observed difference: -0.25 (95% CI: -0.38 to -
0.13)• 5% of tofacitinib patients and 11% of placebo
patients were dropouts• MMRM assumes that missingness is at random
www.fda.gov
![Page 18: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/18.jpg)
18
Applicant’s Sensitivity Analysis• Jump-to-reference multiple imputation• Can also be viewed as having targeted
treatment policy estimand under different missingness mechanism assumption
• Did not comprehensively explore space of plausible, alternative assumptions
www.fda.gov
![Page 19: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/19.jpg)
19
FDA Reviewer’s Sensitivity Analysis
www.fda.gov
![Page 20: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/20.jpg)
20
FDA Reviewer’s Sensitivity Analysis• Proportions of dropouts were low• Even placebo completers did not observe a
mean improvement as large as -0.5• Points at which results tipped were considered
implausible, so evidence of efficacy was convincing despite missing data
www.fda.gov
![Page 21: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/21.jpg)
21
STRENGTHS AND LIMITATIONS4
www.fda.gov
![Page 22: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/22.jpg)
22
Data Use Efficiency• Does not condition on prognostic
characteristics, dropout pattern, and outcomes observed prior to dropout
• Tipping point analysis methods that do so may have gains in efficiency
www.fda.gov
![Page 23: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/23.jpg)
23
Assumption in Benchmark Setting• Benchmark setting: 𝛿𝛿𝑡𝑡 = 𝛿𝛿𝑐𝑐 = 0• Does not assume that missingness depends on
both treatment assignment and other variables– May not be appropriate for a primary analysis– Can differ from assumption made by a primary
analysis such as MMRMwww.fda.gov
![Page 24: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/24.jpg)
24
Simple to Perform• No imputation needed• Calculation of point estimate, test statistic, and
CI are all straightforward
www.fda.gov
![Page 25: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/25.jpg)
25
Minimal Assumptions• Normality of outcomes is not assumed• No particular parametric form is assumed• Allows for analysis of ordered categorical
variables such as HAQ-DI or binary variables such as ACR20
www.fda.gov
![Page 26: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/26.jpg)
26
Sensitivity Parameters are Intuitive• For treatment arm ℎ, 𝛿𝛿ℎ is difference in true
means between dropouts and completers• Ease of interpretation of sensitivity parameters
facilitates cross-disciplinary discussion of sensitivity analysis results
www.fda.gov
![Page 27: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/27.jpg)
27
CLOSING THOUGHTS5
www.fda.gov
![Page 28: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/28.jpg)
28
Key Takeaways• Method serves as valuable tool for sensitivity
analyses, with limited and transparent assumptions
• Method allows for valid statistical inference without the need for imputation
www.fda.gov
![Page 29: A Tipping Point Method to Evaluate Sensitivity to …...A Tipping Point Method to Evaluate Sensitivity to Potential Violations in Missing Data Assumptions Cesar Torres September 24,](https://reader030.vdocuments.mx/reader030/viewer/2022041104/5f04de677e708231d4101d7a/html5/thumbnails/29.jpg)