Slide 1

Evidence-based Guidelines: Looking Back, and Looking Ahead David M Eddy MD PhD December 10, 2012 Slide 2 A change in topic Maximizing Guideline Effectiveness for Individual Patients Slide 3 A change in topic Maximizing Guideline Effectiveness for Individual Patients See Eddy DM, Adler J, Patterson B, Lucas D, Smith KA, Morris M. Individualized guidelines: the potential for increasing quality and reducing costs. Ann Intern Med. 2011;154:627-34 Slide 4 Evidence-based Guidelines: Looking Back, and Looking Ahead Some observations from the last 40 years, and some recommendations for the next 20 A personal perspective The way I see it Loaded with conflicts of interest Slide 5 Observations (early 1970s) Medical decision making is fundamentally flawed How were decisions being made? Lousy evidence Errors in reasoning Gross oversimplifications Wide variations in beliefs We cant trust clinical judgment, the art of medicine, expert opinion, or expert consensus More formal methods are needed Slide 6 My first encounter with lousy evidence: Treatment of ocular hypertension Chance of field defects or blindness Worse Slide 7 Wide variations in beliefs Fifty-eight experts estimates of the chance of a spontaneous rupture of a silicone breast implant 0% 0.2% 0.5% 1% 1% 1% 1.5% 1.5% 2% 3% 3% 4% 5% 5% 5% 5% 5% 5% 5% 6% 6% 6% 8% 10% 10% 10% 10% 13% 13% 15% 15% 18% 20% 20% 20% 25% 25% 25% 30% 30% 40% 50% 50% 50% 62% 70% 73% 75% 75% 75% 75% 80% 80% 80% 80% 80% 80% 100% Slide 8 8 A proposed manifesto (1976-1980) American Cancer Society recommendations for the cancer-related checkup "In making these recommendations, the society has four main concerns: First, there must be good evidence that each test or procedure recommended is medically effective in reducing morbidity or mortality; second, the medical benefits must outweigh the risks; third, the cost of each test or procedure must be reasonable compared to its expected benefits; and finally, the recommended actions must be practical and feasible. Guidelines for the cancer-related checkup: recommendations and rationale. CA-A cancer journal for clinicians, 1980;30:193-240. Slide 9 Evidence Outcomes Decision Scientific Judgment Preference Judgment Compare Benefits, Harms and Costs Analyze Evidence A progression of methods for guidelines (early 1980s) Slide 10 Evidence Outcomes Decision Scientific Judgment Preference Judgment Compare Benefits, Harms and Costs Analyze Evidence Global subjective judgment, or consensus-based Slide 11 11 Evidence Outcomes Decision Scientific Judgment Preference Judgment Compare Benefits, Harms and Costs Analyze Evidence Evidence-based Slide 12 12 EvidenceDecision Scientific Judgment Preference Judgment Compare Benefits, Harms and Costs Analyze Evidence Outcomes-based Outcomes Slide 13 13 Evidence Outcomes Decision Scientific Judgment Preference Judgment Compare Benefits, Harms and Costs Analyze Evidence Preference-based Slide 14 Observations Evidence based guidelines are thriving Carry on! Evidence-based guidelines are not enough They only ask if there is evidence of effectiveness Do not ask magnitude of effectiveness, or harms, or costs Dont enable comparison of benefits, harms and costs Implicitly assume: If there is evidence of effectiveness it should be done All effective treatments are equally important o E.g. all guidelines, all performance measures Increasing use of evidence-based treatments will lower costs Slide 15 Observations We need to move to outcomes-based guidelines Estimate the magnitude of benefits and harms (and costs) Use the information to make tradeoffs, and set priorities We cant accomplish outcomes-based guidelines with clinical trials alone Too many questions Trials require lots of money, time, and agreement to participate Results of trials from one setting cant (shouldnt) be transferred to other settings Results are rapidly outdated Slide 16 16 Trials from one setting cant simply be transferred to other settings Slide 17 17 Trials are statistically homogeneous: no significant differences across the different settings In fact the rates of the outcomes vary widely in the different settings Slide 18 18 Rates of major coronary events in the different settings Slide 19 19 Effects of treatments in the different settings Slide 20 Observation Never use a fixed effects model in a meta-analysis Unless its a multi-institutional trial using identical protocols Slide 21 21 The Q statistic shouldnt be used in meta-analysis; just look at the designs of the trials The calculated effect of the treatment and 95% confidence interval The actual results vary widely and 5 of the 10 trials are on or outside the 95% confidence limit. Slide 22 Observations Outcomes-based medicine requires mathematical models Cant depend on clinical judgment Not feasible to do all the needed clinical trials Have to use models (as in every other field) Use existing evidence to build and validate models Then use the models to answer adjacent questions o Similar populations, settings, interventions, outcomes, The mathematical methods, computing power, and software exist Data are becoming increasingly available Its feasible Slide 23 Observations Mathematical models have not yet proven themselves to be credible sources for decisions Many models have structures and make assumptions that lack face validity Few decision makers have the training needed to understand a model Even those that do, cant run (test) a model in their heads Different models give different answers to same question Very few models are validated against the real world Validations themselves can be dressed up Extremely difficult for decision makers to evaluate a model Slide 24 Observations Mathematical models have not yet proven themselves to be credible sources for decisions Many models have structures and make assumptions that lack face validity Few decision makers have the training needed to understand a model Even those that do, cant run (test) a model in their heads Different models give different answers to same question Very few models are validated against the real world Validations themselves can be dressed up Extremely difficult for decision makers to evaluate a model anyone Slide 25 Different models give different results to same question Slide 26 The effect of a treatment Slide 27 Sensitivity analysis doesnt fix the problem Slide 28 Observations We need to move beyond Markov models The assumptions do not match clinical medicine The patient is in one and only one state at any time Patients jump from one state to another at intervals The chance of jumping from one state to another does not depend on how long the patient has been in the state o Past history, duration of disease, dont matter Markov modelers develop ingenious methods to override the assumptions A clue that we should be using a different model structure Put the modeling talent into developing better structures Slide 29 Observations We need to improve healthcare models Build models that more closely match physiology and clinical practice Provide complete transparency: Make technical and non- technical reports accessible to all serious users Help decision makers understand the descriptions Validate, validate, validate. Show that the model can Accurately calculate rates of important outcomes seen in the real world Accurately calculate effects of interventions seen in the real world Adjust for different settings (populations, care processes, behaviors) Make working copies available for decision makers to explore Slide 30 Example of validation of an outcome: First stroke, males, by age Slide 31 Example of outcome validation: Stroke in Placebo Group of SHEP Slide 32 Observations: Costs There has been almost no progress in explicit consideration of costs The cost taboo The R word We need to attack the cost issue head on Conduct lots of studies of Rationing by patient choice JAMA 1991; 265:105-108 Confirm (or refute) that costs are important, and patients are willing to trade off cost and quality Slide 33 Observations: Cost/QALY Stop using cost/QALY as the reference method for addressing costs Cant trust calculations of either QALYs (clinical outcomes) or costs Cost/QALY calculations assume nothing changes in future E.g., no new science, technology, or evidence Cost/QALY doesnt correspond to anything important in real decisions No one knows what the right level of cost/QALY is Cost/QALY has good academic roots, but is not helpful in reality Slide 34 Observations Instead, calculate what decision makers really want to know Clinical and economic outcomes they understand and care about Time horizons pertinent to their decisions Slide 35 Observations: Guidelines Replace current population-based guidelines with a single, integrated, individualized guideline (Annals, 2011) Span all pertinent conditions Use all important information about the patient Preserve continuous nature of biomarkers and risk factors; no sharp cut points Calculate the risk of all important outcomes Calculate the change in risk with all potential treatments Present the information to the physician and patient They decide Patient centered, preference-based guideline Rank patients by expected benefit, and use ranked list for outreach, incentives, and priority setting Slide 36 Observations: Performance measures Replace current population-based performance measure with a single, integrated, outcomes-based quality measure (Health Affairs, Nov 2012) Based on the expected improvement in clinical outcomes, not processes or treatment targets Captures all pertinent conditions and outcomes Includes everything a provider might do to improve the outcomes of interest Leaves providers free to find most efficient ways to improve outcomes Can also include costs and calculate value Slide 37 A Global Outcomes Score Global outcome score Bone mineral densityCholesterolBlood pressureSmoking Drug A Hip fractures Strokes Heart attacks Lung cancer Drug B Combinations Slide 38 A swan song? The Mute Swan Refers to an ancient Greek belief that the Mute Swan (Cygnus olor) is completely silent during its lifetime until the moment just before death, when it sings one beautiful song Cygnus olor is not mute but lacks bugling call, merely honking, grunting, and hissing on occasion Cygnus olor