solas webinar slides
TRANSCRIPT
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
Visual Sensitivity Analysis
for Missing DataAodhán O’LearyAndrew Grannell
Webinar4th June 2015
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
Overview• Missing Data Analysis & Imputation• Sensitivity Analysis & Tipping Points• SOLAS Tipping Points Demo• Round-up and Q&A
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
Aodhán O’Leary
Statistician
MSc. (Stat.) , University College Cork
Andrew Grannell
Head of Research and Development
MSc. (Stat.), University College Cork
Who we are
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
Missing Data Analysis
• Missing Data Mechanisms (MCAR, MAR, MNAR)
• Complete Case analysis…• LOCF, LVCF, BOCF, BVCF…• Inverse probability weighting• Maximum Likelihood estimation• Multiple Imputation• And others…
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
Missing Data Analysis
• Missing Data Mechanisms (MCAR, MAR, MNAR)
• Complete Case analysis…• LOCF, LVCF, BOCF, BVCF…• Inverse probability weighting• Maximum Likelihood estimation• Multiple Imputation• And others…
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
Imputation Overview
• “We use MI to replace missing values with plausible values in order to exploit the information that was recorded”
- Andridge & Little, 2010
• MI methods can be classed into two main types– Model based methods– Hot deck methods
• Techniques incorporate variability to give a number of imputed values for each missing data point
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
Sensitivity Analysis
• Consists of THREE main steps:1. Drawing conclusions under working
assumptions about missing data,2. Identifying a set of plausible alternative
assumptions,3. Studying the variation in the statistical
output & conclusions under the alternative settings
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
Regulatory Advice on Sensitivity Analysis
• “Examining sensitivity to the assumptions about the missing data mechanism should be a mandatory component of reporting.” [1]
• “A targeted range of…‘sensitivity analyses’ can help to investigate and understand the robustness of estimates…” [2]
• “‘Tipping point’ analysis recommended by NRC appealing for regulators” [3]
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
Regulatory Advice on Sensitivity Analysis
• “This…survey of Japan- and foreign-based pharmaceutical manufacturers revealed that missing data are frequent in confirmatory trials…the most popular method for handling missing data was LOCF imputation…LOCF is not justified in disease areas in which missing data are frequent” [4]
• “One type of supportive analysis might investigate treatment effects according to a range of different estimands.” [1]
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
Treatment Control
Treatment Effect Estimation
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
Treatment Control
Treatment Effect Estimation¿𝑆𝑢𝑐𝑐𝑒𝑠𝑠
¿𝐹𝑎𝑖𝑙𝑢𝑟𝑒
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
Treatment Control
Treatment Effect EstimationTreatment: Control:
¿𝑆𝑢𝑐𝑐𝑒𝑠𝑠
¿𝐹𝑎𝑖𝑙𝑢𝑟𝑒
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
Treatment Control
Treatment Effect EstimationTreatment: Success: Control: Failure:
¿𝑆𝑢𝑐𝑐𝑒𝑠𝑠
¿𝐹𝑎𝑖𝑙𝑢𝑟𝑒
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
Treatment Control
Treatment Effect EstimationTreatment: Success: Control: Failure:
¿𝑆𝑢𝑐𝑐𝑒𝑠𝑠
¿𝐹𝑎𝑖𝑙𝑢𝑟𝑒
𝜏=(∑𝑖=1
𝑁
𝑦 𝑖(1)−∑𝑖=0
𝑁
𝑦 𝑖(0)) /𝑁
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
Treatment Control
Treatment Effect EstimationTreatment: Success: Control: Failure:
¿𝑆𝑢𝑐𝑐𝑒𝑠𝑠
¿𝐹𝑎𝑖𝑙𝑢𝑟𝑒
𝜏=(∑𝑖=1
𝑁
𝑦 𝑖(1)−∑𝑖=0
𝑁
𝑦 𝑖(0)) /𝑁 �̂�=𝑦𝑇− 𝑦𝐶
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
T C T C
Effect Estimation with Missing Data
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
T C T C
Effect Estimation with Missing Data
�̂�=𝑦𝑇− 𝑦𝐶
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
T C T C
Effect Estimation with Missing Data
�̂�=𝑦𝑇− 𝑦𝐶 ~𝜏= 𝑦𝑇𝑁 𝑇+𝑦𝑇𝑁 𝑇
𝑁𝑇 −𝑦𝐶𝑁𝐶+𝑦𝐶𝑁𝐶
𝑁𝐶 obs obs obs obs mis mis mis mis
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
Tipping Point Plotting
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
Tipping Point Plotting
¿
¿
Control Group
Treatment Group
Worst Case
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
Tipping Point Plotting
¿
¿
Control Group
Treatment Group
¿
¿
Worst Case Best Case
Control Group
Treatment Group
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
Enhanced TP Plots Background
• Uses the above plots as the basis for analysing missing data.
• Includes the data from imputations.• Makes it possible to compare the results
of analysis under multiple assumptions.• Gives a more detailed visual analysis,
and allows you to compare effect estimates.
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
Constructing Enhanced TP Plots
1. Get # of missing outcomes for Treatment & Control group (outline x and y axis range),
Set Up the Axes
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
1. Get # of missing outcomes for Treatment & Control group (outline x and y axis range),
2. Potential # of success in Treatment group on x-axis and potential # of success in Control on y-axis,
Constructing Enhanced TP Plots
Set Up the Axes
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
1. Get # of missing outcomes for Treatment & Control group (outline x and y axis range),
2. Potential # of success in Treatment group on x-axis and potential # of success in Control on y-axis,
3. Calculate estimated treatment effect () and p-values for a given hypothesis for each cell,
Constructing Enhanced TP Plots
Set Up the Axes
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
1. Get # of missing outcomes for Treatment & Control group (outline x and y axis range),
2. Potential # of success in Treatment group on x-axis and potential # of success in Control on y-axis,
3. Calculate estimated treatment effect () and p-values for a given hypothesis for each cell,
4. Set up grid to highlight all combinations that result in rejecting the null hypothesis,
Constructing Enhanced TP Plots
Draw the Tipping Point ‘Staircase’
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
1. Get # of missing outcomes for Treatment & Control group (outline x and y axis range),
2. Potential # of success in Treatment group on x-axis and potential # of success in Control on y-axis,
3. Calculate estimated treatment effect () and p-values for a given hypothesis for each cell,
4. Set up grid to highlight all combinations that result in rejecting the null hypothesis,
5. Overlay imputation information (squares, contours)
Constructing Enhanced TP Plots
Overlay Imputation Info
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
Enhanced Tipping Point Plotting
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
Overview of SOLAS
• Stand alone Missing data package• FIVE Multiple Imputation techniques• Drag & Drop interface• SOLAS syntax language feature• Plotting & Visualisation techniques
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
ETP Plots - 1000 Imputations
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
ETP Plots - 1000 Imputations
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
ETP Plots - 1000 Imputations
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
ETP Plots - 1000 Imputations
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
ETP Plots - 1000 Imputations
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
ETP Plots - 1000 Imputations
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
SOLAS Tipping Points Demo
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
Round Up
• Background & History
• Overview of Missing Data analysis, Imputation and Tipping Points
• SOLAS Demo
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
ICH Expert Group Survey
• The purpose of this survey is to gather data on related current practices, primarily focused on clinical trials involving drugs, vaccines and biologics. The survey is being distributed to various regions within ICH, including industry and regulatory communities. If a question is not applicable to your job role, please select “I am unable to answer this question/not applicable to my job role”.
• The ICH E9 Revision 1 working group thanks you in advance for your participation. Please complete the survey by Friday 5th June 2015.
• Link to survey: https://www.surveymonkey.com/r/VPMV57Z
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
Questions?
Thank you
ICH Survey: https://www.surveymonkey.com/r/VPMV57Z
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
References
1. NRC-Panel (2010) The Prevention and Treatment of Missing Data in Clinical Trials. National Academies Press.
2. ICH E9(R1) (2014) Addendum to Statistical Principles of Clinical Trials. CPMP/ICH/363/96.
3. LaVange, L.M. (February 2015) Missing Data Issues in Regulatory Clinical Trials. Symposium conducted at the meeting of the Japanese Pharmaceutical Manufacturers Association.
Statistical Solutions | Registered in Ireland, Reg. No 233638 | www.statsols.com
References
4. Tanaka, S., Fukinbara, S., Tsuchiya, S., Suganami, H., and Ito, Y.M. (2014). Current Practices in Japan for the Prevention and Treatment of Missing Data in Confirmatory Clinical Trials: A Survey of Japanese and Foreign Pharmaceutical Manufacturers.Therapeutic Innovation & Regulatory Science, 48(6) 717-723.
5. Campbell, G., Pennello, G., and Yue, L. (2011). Missing Data in the Regulation of Medical Devices. Journal of Biopharmaceutical Statistics, 21(2):180–195.
6. Victoria Liublinska and Donald B. Rubin (2014). Sensitivity analysis for a partially missing binary outcome in a two-arm randomized clinical trial. Statistics in Medicine, 33(24):4170-4185.