wishful thinking vs. causal analysis: the case of fine particulate matter (pm2.5) health effects...

Wishful thinking vs. causal analysis: The case of fine particulate matter (PM2.5) health effects Tony Cox, Douglas Popken [email protected] September 19, 2012

Four main points 1.Prevalent dramatic claims: Reducing PM2.5 extends lives! (EPA, WHO, others) 2.No objective basis for such causal claims Association-based, opinion-driven No objective, reproducible causal analysis 3.More objective methods for causal analysis are available (from AI, econometrics, etc.) New to air pollution health effects research 4.Causal analysis: Reducing PM2.5 has no detectable impact on extending lives. 2

Dramatic causal claims 3 www.catf.us/resources/publications/view/159

4 What Can Be Done to Slow Climate Change? ScienceDaily (Jan. 12, 2012) A new study led by a NASA scientist highlights 14 key air pollution control measures that, if implemented, could slow the pace of global warming, improve health and boost agricultural production. The research, led by Drew Shindell of NASA's Goddard Institute for Space Studies (GISS) in New York City, finds that focusing on these measures could slow mean global warming 0.9 F (0.5C) by 2050, increase global crop yields by up to 135 million metric tons per season and prevent hundreds of thousands of premature deaths each year. We've shown that implementing specific practical emissions reductions chosen to maximize climate benefits would also have important 'win-win' benefits for human health and agriculture," said Shindell. http://www.sciencedaily.com/releases/2012/01/120112193442.htm

5 Air Pollutant Levels of Particulates and Ozone Add to Public Health Burden WASHINGTON, Feb. 7, 2012 /PRNewswire-USNewswire/ Despite reductions in US air pollution over the past several decades, resulting from the combined efforts of government, nonprofit and industrial sectors, concentrations of both fine particles and ozone remain a public health concern, particularly in urban areas. A new study by economists, scientists, and modelers at the U.S. Environmental Protection Agency (EPA) shows that levels of fine particulate matter (PM2.5) and ozone (O3) pose significant risks to public health, including an increased risk of mortality. These findings could be used to inform policy decisions about approaches to reducing key pollutants and thereby improve health outcomes, particularly in urban areas. The analysis, entitled "Estimating the National Public Health Burden Associated with Exposure to Ambient PM2.5 and Ozone," was co-authored by EPA's Neal Fann, Amy Lamson, Susan Anenberg, Karen Wesson, David Risley, and Bryan Hubbell. Their research findings are published in the January 2012 issue of the journal Risk Analysis, published by the Society for Risk Analysis.

Claims are unlikely to be true: Pr(true association) < 1/2 (maybe < 0.05) true = not spurious, due to confounding or modeling choices, uncertainties and biases Can investigate with BMA, nonparametrics Daily temperature is a strong confounder Pr(association is causal | true) < 1/2 Granger tests, SEM tests, BN tests, etc. Pr(no relevant threshold | true) < 1/2 Adjustment factor for trends < 1/2 Combining, Pr(no effect) > 90% 6

7 http://www.scientificintegrityinstitute.org/JESEE2005.pdf How sure is association?

8 8 Ambiguous associations of PM 2.5 with all-cause mortality Why so many negatives? EPAs resolution: Assume 100% probability for positive effects, 0% for negative. Rationale: Confidence.

9 http://www.scientificintegrityinstitute.org/JESEE2005.pdf How sure is association? Reported significant associations disappear when model uncertainty is accounted for by BMA (Clyde, Koop & Tole) Discrepancies not resolved

Claims are unlikely to be true: Pr(true association) < 1/2 (maybe < 0.05) true = not spurious, due to confounding or modeling choices, uncertainties and biases Can investigate with BMA, nonparametrics Daily temperature is a strong confounder Pr(association is causal | true) < 1/2 Granger tests, SEM tests, BN tests, etc. Pr(no relevant threshold | true) < 1/2 Adjustment factor for trends < 1/2 Combining, Pr(no effect) > 90% 10

11 http://www.sciencedirect.com/science/article/pii/S0890623810003011 WHO explains how to calculate burden of disease: Assume the conclusion, that exposure causes all differences in risks between more- and less-exposed groups Thoroughly confuses association with causation Widely accepted/applied, yet yields mistaken conclusions

Population attributable fraction WHO Global Burden of Disease studies misinterpret association as causation 12 http://www.who.int/healthinfo/global_burden_disease/GlobalHealthRisks_report_annex.pdf Wrong

But PAF only indicates association! Like relative risk (RR), PAF depends only on association. PAF > 0 if exposed people have higher risk. 13 http://www.who.int/healthinfo/global_burden_disease/GlobalHealthRisks_report_annex.pdf

14 A B PAF > 0 So, reducing exposure will reduce risk The PAF mental model exposure risk

15 A (old) B (young) Comparing levels of two variables says nothing about how changing one changes the other Baby aspirin & heart attack risk exposure risk

16 A B Comparing levels of two variables says nothing about how changing one changes the other Baby aspirin & heart attack risk exposure risk ?

17 A B Comparing levels of two variables says nothing about how changing one changes the other Baby aspirin & heart attack risk exposure risk

What we want to know Would further reducing PM2.5 really increase life expectancy? Not by assumption, but in fact Have past reductions in PM2.5 truly increased life expectancy or reduced mortality/morbidity? If so, how large are these effects? What fraction of PAF is causal? How can we know? How sure can we be? 19

How not to get trustworthy answers Regress mortality rates against exposure and other (e.g., weather) variables. Interpret coefficients causally Ask experts whether they think statistical associations are causal Compare mortality rates pre and post reduction in PM2.5. Interpret any reduction causally. Biological hand-waving 20

How not to get trustworthy answers Regress mortality rates against exposure and other (e.g., weather) variables. Interpret coefficients causally Ask experts whether they think statistical associations are causal Compare mortality rates pre and post reduction in PM2.5. Interpret any reduction causally. Biological hand-waving 21 = Current EPA methodology

Summary of EPA approach so far Show that associations (might) exist Always possible with bad statistics, especially when strong confounder is present Treat as causal, based on expert judgment Do not test causal hypothesis objectively Acknowledge verbally that association might not necessarily be causal but treat it as causal for quantitative claims/policy Precautionary, responsible, agency discretion 22

Summary of EPA main arguments 1.Burden of mortality calculations 2.Claimed robust epidemiological associations 3.Intervention studies (Dublin) 4.Many other associations 5.Claimed biological plausibility 6.Exposure-response (steepest at origin) 7.Claimed big benefit:cost ratio for regulation 23

Summary of main refutations 1.Mortality burden calculations: Misuse PAFs 2.Claimed epidemiological associations: Highly uncertain. (True, but not effective so far.) 3.Intervention studies (Dublin): Misuse trends 4.Other associations: Speculative 5.Claimed biological plausibility: No; threshold 6.Exposure-response: Model does not fit data 7.Benefit-cost: Ignores discrete uncertainties 24

Example: Intervention study 25

Example: Intervention study 26

Cause and effect? 27 If mortality rate was decreasing at least as quickly before the intervention (coal ban) as after... Then the time series provide no evidence at all that the ban affected mortality!

Did the ban stop progress? 28 Informal causal conclusions are just subjective opinions, with no known validity.

Did the ban stop progress? 29 Formal methods are available (e.g., intervention analysis, Box- Tiao analysis, change point detection) for testing whether a time series changed at the time of an intervention. Not used in the Dublin study. Informal causal conclusions have no known validity.

Post hoc fallacy 30 http://ccgi.newbery1.plus.com/blog/?p=220&cp=all

31 Inhal Toxicol.Inhal Toxicol. 2007 Apr;19(4):343-50. The big ban on bituminous coal sales revisited: serious epidemics and pronounced trends feign excess mortality previously attributed to heavy black-smoke exposure. Wittmaack KWittmaack K. SF-National Research Centre for Environment and Health, Institute of Radiation Protection, Neuherberg, Germany. [email protected] Abstract The effect of banning bituminous coal sales on the black-smoke concentration and the mortality rates in Dublin, Ireland, has been analyzed recently. Based on the application of standard epidemiological procedures, the authors concluded that, as a result of the ban, the total nontrauma death rate was reduced strongly (-8.0% unadjusted, -5.7% adjusted). The purpose of this study was to reanalyze the original data with the aim of clarifying the three most important aspects of the study, (a) the effect of epidemics, (b) the trends in mortality rates due to advances in public health care, and (c) the correlation between mortality rates and black-smoke concentrations. Particular attention has been devoted to a detailed evaluation of the time dependence of mortality rates, stratified by season. Death rates were found to be strongly enhanced during three severe pre-ban winter-spring epidemics. The cardiovascular mortality rates exhibited a continuous decrease over the whole study period, in general accordance with trends in the rest of Ireland. These two effects can fully account for the previously identified apparent correlation between reduced mortality and the very pronounced ban-related lowering of the black- smoke concentration. The third important finding was that in nonepidemic pre-ban seasons even large changes in the concentration of black smoke had no detectable effect on mortality rates.

Old arguments against causal association True, but not persuasive to EPA: Associations unclear Depend on modeling assumptions, some of which are false Unmodeled errors and heterogeneity in exposures Disappear when model uncertainty is accounted for via Bayesian Model Averaging (BMA) Associations weak, inconsistent Associations are not causation 32

33 Model selection biases Confounding Temperature Co-pollutants

34 Weak association Confounding by weather, co- pollutants Exposure errors Inconsistencies Small effect No experimental confirmation

35 Lack of coherence Causality not demonstrated

Technical issues for causality Association is not causation (!) Strong, biologically plausible, specific, temporal, specific association is still not causation! Regression coefficients causal coefficients After does not imply because of Post hoc fallacy Must rule out other (non-causal) explanations Model selection, specification, and uncertainty; Omitted confounders, errors in exposure, etc. Natural variability, regression to mean, trends 36

Such arguments have been inconclusive 37

Needed: Causal revolution! Correct causal analysis can and should revolutionize our understanding of air pollution health effects and must do so, to support better science, policy-making, and decisions 38

Changing the technical game Show that there is no association, after conditioning on non-pollution causes (e.g., temperature) Show that there is no evidence of a causal relation between PM2.5 and mortality rate (despite large power to detect if there) 39

Four main points 1.Prevalent dramatic claims: Reducing PM2.5 extends lives! (EPA, WHO, others) 2.No objective basis for such causal claims Association-based, opinion-driven No objective, reproducible causal analysis 3.More objective methods for causal analysis are well established in other disciplines New to air pollution health effects research 4.Causal analysis: Reducing PM2.5 has no detectable impact on extending lives. 40

What we want to know Causal analysis tells us: Would further reducing PM2.5 really increase life expectancy? Have past reductions in PM2.5 truly increased life expectancy or reduced mortality/morbidity? If so, how large are these effects? How can we know? How sure can we be? 41

How to get trustworthy answers Causal analysis methods: Path analysis Structural equation models (SEM) Marginal structural models (MSM) (counterfactual) Bayesian network causal models Panel data analysis of changes Quasi-experimental design & analysis Interrupted time series analysis Granger causality tests Learning dynamic causal models from data 42

Doing better: Cross-sectional data Simon-Blalock tests Structural equations models Path analysis LISREL, CALIS, AMOS, EQS Bayesian networks Conditional independence tests 43

Doing better: Longitudinal data Panel data analysis Look at change in X and subsequent change in Y, instead of at associations between levels of X and levels of Y 44

Doing better: Intervention causality tests Box-Tiao intervention analysis Change-point detection Hidden Markov Models System identification Quasi-experiments 45

How to do causal inference right True causal relations cannot be explained away by other variables (e.g., confounders) Conditional independence tests True causal relations help to correctly predict how changes in inputs will be followed by changes in outputs Granger test Associations of past levels do not imply this True causal relations can be composed Chains, SEMs, BNs, causal graphical models 46

Doing better Lets apply objective tests for possible causation! 47

Four main points 1.Prevalent dramatic claims: Reducing PM2.5 extends lives! (EPA, WHO, others) 2.No objective basis for such causal claims Association-based, opinion-driven No objective, reproducible causal analysis 3.More objective methods for causal analysis are well established in other disciplines New to air pollution health effects research 4.Causal analysis: Reducing PM2.5 has no detectable impact on extending lives. 48

Applying better methods to Hundred Cities data: No causal impact of PM2.5 on mortality (Preview) 49

Why no positive association? 50

Warmer days have lower mortality rates 51

Higher PM2.5 (by city, month, and year) has higher mortality rates: Positive association 52

But, higher PM2.5 (by city and year) does not have higher mortality rates 53 Implication: Positive association arises from months Cold winter months explain the positive association.

Increases in temperature reduce mortality risks (1999-2000, 100 cities) 54 Comparing changes over time is the essence of panel data analysis.

Increases in PM2.5 do not increase mortality risks 55

Results of Granger causality tests: Temperature Granger-causes deaths. PM2.5 does not. 56

Summary of findings NMMAPS data (100 Cities) shows that: Temperature explains the association between PM2.5 and mortality Statistical smoothing/seasonal adjustment obscures, but does not remove, confounding PM2.5 does not cause any detectable increase in mortality rate Cold temperature increases mortality rates (and PM2.5) 57

Four main points: So what? 1.Prevalent dramatic claims: Reducing PM2.5 extends lives! (EPA, WHO, others) 2.No objective basis for such causal claims Association-based, opinion-driven No objective, reproducible causal analysis 3.More objective methods for causal analysis are well established in other disciplines New to air pollution health effects research 4.Causal analysis: Reducing PM2.5 has no detectable impact on extending lives. 58

wishful thinking vs. causal analysis: the case of fine particulate matter (pm2.5) health effects...

Documents