BARRIERS TO A MORE EVEN-HANDED BARRIERS TO A MORE EVEN-HANDED TREATMENT OF UNCERTAINTIES IN TREATMENT OF UNCERTAINTIES IN

PROJECTED ECONOMIC COSTS AND HEALTH PROJECTED ECONOMIC COSTS AND HEALTH BENEFITS OF ENVIRONMENTAL HEALTH BENEFITS OF ENVIRONMENTAL HEALTH

REGULATIONS REGULATIONS

DALE HATTISDALE HATTISCLARK UNIVERSITYCLARK UNIVERSITY

Approach to the Problem• Adam has introduced the issues and prior

observations of imbalance in the quantification of uncertainties in costs vs benefits.

• I chose to first build additional observations from the most recent available EPA RIAs.

• This gave me a basis to interview participants in creating these RIAs and discuss ideas about why they were crafted as the were.

• Today I will first build some historical context for “impact analysis” before moving to the observations from recent RIAs.

Historical Background of “Impact Analysis”--

• Which “Facts” need to be assembled for value choices to be made?

• Corps of Engineers tradition of traditional cost benefit analysis (promoting societal aggregate net benefit as the goal)

• Environmental movement of Late 60s-early 70s aimed to right distributional wrongs

• NEPA (1969) – Required analysis of environmental impacts not previously

assessed.– Allowed enforcement by citizen suits.– Did not change the underlying decision-making mandate

of affected agencies.

Political Opponents of Environmentalists Borrow a Page

• 1974—Ford “Whip Inflation Now (WIN)” program institutes requirement for “Inflation Impact Statements”

• Requirement is based on an Executive Order, not a congressional law, so enforcement is not via the courts but solely by administrative appeal to the Office of Management and Budget

• For both kinds of impact statements, advocates were not powerful enough to change underlying legislative mandates, but sought impact statements as a way of less formally changing the mix of considerations influencing administrative choices.

• Statement initially required to say whether the costs are “justified” by the benefits, not the stronger statement (later used under Regan) that benefits “outweigh” costs.

Development of Mandates for Uncertainty Analysis

• Risk analysis emerges as a discipline beginning in the mid-late 1970s and early 1980s.

• Agencies develop “conservative” procedures for estimating risks and associated risk reduction benefits—simplifying assumptions thought likely to overstate risks and therefore lead to protective standards—placing the burden of uncertainty on protected parties.

• Such “conservative” estimation procedures, however did not correspond to the “risk neutral” arithmetic mean “expected value” estimates called for under traditional cost benefit criteria.

• (They may also not have been as health-protective as the participants may have imagined or sold to managers).

Take-away from this background• This discussion sets the policy context for

understanding the role of impact analyses in decision making. The impact analysis advocates did not have the power to change the basic legislative mandates, but did have enough influence to at least get their considerations analyzed and disclosed. This helps to understand why the mandated analyses were never placed at the very center of the decision-making processes and made highly rigorous, but tended to be conducted more with a goal to satisfice (be sufficient to appease) potential opponents after important policy choices had already been made

Bush 1991 “Regulatory Program of the United States”

• Unfortunately, risk-assessment practices continue to rely on conservative models and assumptions that effectively intermingle important policy judgments within the scientific assessment of risk. Policymakers must make decisions based on risk assessments in which scientific findings cannot be readily differentiated from embedded policy judgments. This policy environment makes it difficult to discern serious hazards from trivial ones, and distorts the ordering of the Government’s regulatory priorities. In some cases, the distortion of priorities may actually increase health and safety risks.

OMB Circular A-4 (2003)

• “When benefit and cost estimates are uncertain (for more on this see "Treatment of Uncertainty" below), you should report benefit and cost estimates (including benefits of risk reductions) that reflect the full probability distribution of potential consequences. Where possible, present probability distributions of benefits and costs and include the upper and lower bound estimates as complements to central tendency and other estimates.”

Obama (2011)

• “The goal of the agency’s uncertainty analysis is to present both a central “best estimate,” which reflects the expected value of the benefits and costs of the rule, as well as a description of the ranges of plausible values for benefits, costs, and net benefits, which informs decision-makers and the public of the degree of uncertainty associated with the regulatory decision.”

New Observations

Example 2

Example 3

Example 4Rule Date Office Goal Statement Uncertainties in

Benefits Analyzed?

Uncertainties in Costs Analyzed?

USEPA. National emission standards for hazardous air pollutants: Mercury emissions from mercury cell chlor-alkali plants--signed draft federal register notice, 2011(45)

2011 Air National emission standards for hazardous air pollutants (NESHAP) for mercury emissions from mercury cell chlor-alkali plants in response to a petition for reconsideration filed by the Natural Resources Defense Council (NRDC). This action proposes two options. The first option would require the elimination of mercury emissions and thus encourage the conversion to non-mercury technology. The second option would require the measures proposed in 2008 but would still allow chlor-alkali plants to operate..

Yes—derived from particle related mortality reduction co-benefits with monetization. The primary mercury-related benefits are not estimated “due to data, time, and resource limitations”. “We estimate the monetized energy co-benefits of the non-mercury technology option to be $22 million to $43 million (2007$, 3 percent discount rate) in the implementation year (2013). The monetized co-benefits of the regulatory action at a 7 percent discount rate are $14 million to $33 million (2007$). Higher or lower co-benefits estimates are plausible using other assumptions.

Cost benefit summary table presents only a single point estimate of costs. However there is appreciable uncertainty analysis in costs traceable to challenges to the cost analysis from both environmental groups petitioning for the change and industry. “In this revised analysis, we recognize that there are significant uncertainties in estimating these costs, and consider ranges of the potential costs (and savings) associated with each cost element. For each element, we do select a “best estimate” to allow the estimation of capital and annual costs of conversion for each facility... .Using more conservative assumptions, our best estimate is that the average annual costs would be between $800,000 and $7 million per year per plant.

Summary Results from New Observations• Overall 9 of the 14 new cases had clearly greater depth of analysis

of uncertainties in benefits than in costs—in accord with the prior findings.

• Of the 4 cases where we judge the treatment of uncertainties in costs and benefits to be reasonably comparable, three represent cases where there was little or no quantitative treatment of uncertainties at all—impact estimates, when given, were single points, often derived using “conservative” thought likely to overstate health/biological benefits.

• In the remaining case of comparable uncertainty treatment, the treatment of cost uncertainties seems to reflect effects of controversy arising from challenges by both environmentalists and industry to a baseline estimate originally derived by/for the agency.

• A final case of arguably greater analysis of cost uncertainty was the adoption of general drinking water standards for use in air planes—where there was some analysis of uncertainty in costs but quantitative estimation of benefits (and their uncertainty) was not attempted

Results of Interviews

• Perceptions of a key EPA analyst– engineering cost estimates don’t come naturally

attached to uncertainty statement, unlike the epidemiologically-based confidence limits for the mortality estimates and other effects associated with fine particle exposures.

– overall the most important source of uncertainty is probably the changing baseline where effects both positive and negative occur far into the future (e.g. 2020 for the NAAQS particulate standard).

Another Interview• Perceptions of 2001-6 OMB/OIRA head John

Graham:– EPA economists, while willing to use statistical

confidence ranges as measures of uncertainty, were very reluctant to go beyond this to potentially more extensive quantitative characterizations of the uncertainties in analyses of costs and benefits.

– EPA economists were comfortable with sensitivity analyses, and happy to illustrate results of different scenarios, but did not recognize the appropriateness of non-statistical analyses and necessarily subjective weightings of relative likelihood of different scenarios.

– Economists’ position on this was reinforced by the general unwillingness of EPA lawyers to sanction the explicit incorporation of numerical uncertainties in official documents such as RIAs.

Speculation on Lawyers’ Reasons• Adam’s perception: I think laypeople (lawyers, judges) think

they understand economics, but are more willing to admit they aren't scientists-- and hence they see an error bar for science and see it as reasonable, whereas an error bar for a cost figure connotes to them "go back and come up with the right answer; you are being lazy to stop with the range."

• Additional possibility: lawyers want to preserve the traditional legal doctrine that courts should show “deference” to the judgments of expert executive agencies within the agencies’ field of expertise. Stating uncertainties in scientifically-quantified health benefits may be judged to disturb this tradition less than more accessible conclusions with respect to costs.

Interpretation• It is clear that the EPA analysts are very aware and cite

in their documents the record of ex-ante vs ex post predictions of costs, and the empirical data on learning curve effects from the innovation/business literature. Learning curve effects resulting in reduced cost estimates have been incorporated into regulatory impact analyses for some mobile source emission rules. However the EPA analysis team seems to have been generally reluctant to draw on non-case-specific information to quantify Bayesian “prior” expectations for the bias and uncertainty in the engineering estimates of compliance costs that they receive.

Positivism vs Critical Realism—Role of Theory/”Metaphysics”

• Positivism—Theory is to be avoided wherever possible in favor of simple analysis/summarization of the data. (Role of statistical analysis is to describe/summarize data.)

• Critical Realism—building a picture of underlying causal relationships—some of which may not be directly observable--is the essence of the goal for scientific research.

Frequentist vs Bayesian Concepts of Probability

• Frequentist—Probability statements can only be made by sampling from some data set; there is no information relevant to an analysis beyond the data.

• Bayesian—Probability is a degree of belief or perception of the likelihood of an event or circumstance, considering all available information.

• Contrast—implications of finding an alligator in the Charles River (near Boston) vs a cockroach in Aunt Suzie’s kitchen.

Conclusion• The most likely principal interpretation for the

economists’ behavior is a frequentist orientation to statistical analysis—case specific information is required for quantitative uncertainty analysis.

• The record of ex-ante vs ex-post comparisons of cost estimates could provide a reasonable starting point for understanding the direction and magnitude of likely errors in cost estimation. Moreover, distributional uncertainty estimates specific to particular cases could be constructed by quantitatively analyzing the past record for the effects of known likely causal determinants of systematic and random errors.

Likely Gains From Improved Uncertainty Analysis of Both Costs and Benefits

• Use distributions to replace the single point “uncertainty” factors now used to adjust estimates of “acceptable daily intakes” and “reference doses” from primary observations for both cancer and non-cancer effects. However this requires setting explicit quantitative goals for RfDs in terms of residual incidence and severity of effects with a particular degree of confidence/uncertainty

• Provides a long term incentive for improvement of the estimates and accountability for the estimators.