using our brains to develop better policy

7
Risk Analysis, Vol. 32, No. 3, 2012 DOI: 10.1111/j.1539-6924.2011.01683.x Perspective Using Our Brains to Develop Better Policy Igor Linkov, 1, Susan Cormier, 2 Joshua Gold, 3 F. Kyle Satterstrom, 4 and Todd Bridges 1 Current governmental practices often use a method called weight of evidence (WoE) to inte- grate and weigh different sources of information in the process of reaching a decision. Recent advances in cognitive neuroscience have identified WoE-like processes in the brain, and we believe that these advances have the potential to improve current decision-making practices. In this article, we describe five specific areas where knowledge emerging from cognitive neu- roscience may be applied to the challenges confronting decisionmakers who manage risks: (1) quantifying evidence, (2) comparing the value of different sources of evidence, (3) reach- ing a decision, (4) illuminating the role of subjectivity, and (5) adapting to new information. We believe that the brain is an appropriate model for structuring decision-making processes because the brain’s network is designed for complex, flexible decision making, and because policy decisions that must ultimately depend on human judgment will be best served by meth- ods that complement human abilities. Future discoveries in cognitive neuroscience will likely bring further applications to decision practice. KEY WORDS: Environmental policy; neuroeconomics; risk analysis 1. INTRODUCTION The Obama administration’s call for a “strat- egy for restoring scientific integrity to government decision making” (1) has affected policy making, es- pecially for politically charged issues such as en- vironmental disasters, energy policy, and emerging technologies. However, better policies for making the scientific process clear and ensuring that quality science truly informs decisions requires better gov- 1 U.S. Army Engineer Research & Development Center, 3909 Halls Ferry Rd., Vicksburg, MS 39180, USA. 2 U.S. Environmental Protection Agency, National Center for Environmental Assessment, 26 W. M. L. King Drive, Cincinnati, OH 45268, USA. 3 Department of Neuroscience, University of Pennsylvania, Philadelphia, PA 19104–6074, USA. 4 Harvard University School of Engineering and Applied Sciences, 29 Oxford St., Cambridge, MA 02138, USA. Address correspondence to Igor Linkov, U.S. Army Engineer Research & Development Center, 696 Virginia Rd., Con- cord, MA 02446, USA; tel: (617) 233–9869; Igor.Linkov@usace. army.mil. ernment processes for translating data into decisions. We believe that recent insights from the field of cog- nitive neuroscience can support and potentially im- prove existing decision-making processes and prac- tices, including informing the development of new tools and models in the area of prescriptive and ap- plied decision-making theory. Current decision practices often use a method called weight of evidence (WoE) to describe how cer- tain data bear on the alternatives under considera- tion. (26) For a choice between two alternatives, the WoE may be defined numerically as the logarithm of the ratio of the probabilities of obtaining the par- ticular data conditioned on each of the two alterna- tives being the true state of the world. More intu- itively, WoE quantifies whether the data were more likely given one alternative or the other and thus can be used to choose the alternative that is most compatible with what has been observed. For exam- ple, WoE methods are key components of ecologi- cal and human health risk assessments, (7) including identification of causes of harm, assessment of risks 374 0272-4332/12/0100-0374$22.00/1 C 2011 Society for Risk Analysis

Upload: igor-linkov

Post on 15-Jul-2016

221 views

Category:

Documents


3 download

TRANSCRIPT

Page 1: Using Our Brains to Develop Better Policy

Risk Analysis, Vol. 32, No. 3, 2012 DOI: 10.1111/j.1539-6924.2011.01683.x

Perspective

Using Our Brains to Develop Better Policy

Igor Linkov,1,∗ Susan Cormier,2 Joshua Gold,3 F. Kyle Satterstrom,4 and Todd Bridges1

Current governmental practices often use a method called weight of evidence (WoE) to inte-grate and weigh different sources of information in the process of reaching a decision. Recentadvances in cognitive neuroscience have identified WoE-like processes in the brain, and webelieve that these advances have the potential to improve current decision-making practices.In this article, we describe five specific areas where knowledge emerging from cognitive neu-roscience may be applied to the challenges confronting decisionmakers who manage risks:(1) quantifying evidence, (2) comparing the value of different sources of evidence, (3) reach-ing a decision, (4) illuminating the role of subjectivity, and (5) adapting to new information.We believe that the brain is an appropriate model for structuring decision-making processesbecause the brain’s network is designed for complex, flexible decision making, and becausepolicy decisions that must ultimately depend on human judgment will be best served by meth-ods that complement human abilities. Future discoveries in cognitive neuroscience will likelybring further applications to decision practice.

KEY WORDS: Environmental policy; neuroeconomics; risk analysis

1. INTRODUCTION

The Obama administration’s call for a “strat-egy for restoring scientific integrity to governmentdecision making”(1) has affected policy making, es-pecially for politically charged issues such as en-vironmental disasters, energy policy, and emergingtechnologies. However, better policies for makingthe scientific process clear and ensuring that qualityscience truly informs decisions requires better gov-

1U.S. Army Engineer Research & Development Center, 3909Halls Ferry Rd., Vicksburg, MS 39180, USA.

2U.S. Environmental Protection Agency, National Center forEnvironmental Assessment, 26 W. M. L. King Drive, Cincinnati,OH 45268, USA.

3Department of Neuroscience, University of Pennsylvania,Philadelphia, PA 19104–6074, USA.

4Harvard University School of Engineering and Applied Sciences,29 Oxford St., Cambridge, MA 02138, USA.

∗Address correspondence to Igor Linkov, U.S. Army EngineerResearch & Development Center, 696 Virginia Rd., Con-cord, MA 02446, USA; tel: (617) 233–9869; [email protected].

ernment processes for translating data into decisions.We believe that recent insights from the field of cog-nitive neuroscience can support and potentially im-prove existing decision-making processes and prac-tices, including informing the development of newtools and models in the area of prescriptive and ap-plied decision-making theory.

Current decision practices often use a methodcalled weight of evidence (WoE) to describe how cer-tain data bear on the alternatives under considera-tion.(2−6) For a choice between two alternatives, theWoE may be defined numerically as the logarithmof the ratio of the probabilities of obtaining the par-ticular data conditioned on each of the two alterna-tives being the true state of the world. More intu-itively, WoE quantifies whether the data were morelikely given one alternative or the other and thuscan be used to choose the alternative that is mostcompatible with what has been observed. For exam-ple, WoE methods are key components of ecologi-cal and human health risk assessments,(7) includingidentification of causes of harm, assessment of risks

374 0272-4332/12/0100-0374$22.00/1 C© 2011 Society for Risk Analysis

Page 2: Using Our Brains to Develop Better Policy

Perspective 375

associated with site contamination or new chemicals,selection of criteria, benchmarks, and permit levels,and prediction of outcomes from remedial interven-tions. However, to date most assessments and subse-quent regulatory decisions based on WoE have beenqualitative in nature,(8,9) reflecting the difficulty ofapplying a rigorous WoE methodology.

The development of WoE, along with the restof assessment and decision theory, has progressedin a state of relative independence from research oncognitive decision making. Nevertheless, these fieldshave converged on several common ideas, in part be-cause of the similarity between the manner in whichsome regions of the brain and explicit mathematicalformulations of WoE weigh evidence.(10) Therefore,we suggest that decision-making approaches in thereal world might be improved by consciously incor-porating principles gleaned from knowledge of howthe brain implements WoE-like metrics.

In this article, we describe five areas whereknowledge emerging from cognitive neuroscience in-tersects with the challenges confronting decision-makers who manage risks: (1) quantifying evidence,(2) comparing the value of different sources of ev-idence, (3) reaching a decision, (4) illuminating therole of subjectivity, and (5) adapting to new infor-mation. Even though the focus of these examplesis on environmental policy, the overall conclusionsare relevant to other policy and risk managementsettings.

2. APPLICATION OF COGNITIVENEUROSCIENCE TO DECISION MAKING

2.1. Quantifying Evidence

An important role for any decisionmaker is todetermine the extent to which a given piece of datarepresents evidence that supports or opposes a par-ticular decision. Recent studies of simple perceptualdecisions (e.g., Is a sensory stimulus present? Whichdirection is it moving?) suggest that under someconditions, the brain appears to transform incomingsensory data into a WoE-like quantity—a compari-son of the conditional probabilities that those datawould have been obtained given either of the twopossible alternatives.(10) These perceptual decision-making findings put brain mechanisms responsiblefor simple perceptual decisions into the framework ofstatistical decision theory, which, for example, showsthat decisions based on this WoE-like aggregation

procedure can optimally balance speed and accuracywhen uncertainty is present.

A striking real-world example of the usefulnessof a WoE-based approach occurred during WorldWar II, when British codebreakers, notably AlanTuring, used it to help crack the Enigma code used bythe Germans. The Enigma was a complicated elec-tromechanical machine used for encrypting and de-crypting secret messages. Part of its complexity arosefrom the fact that the code itself—reflecting cer-tain machine settings—could change daily. Thus, thecodebreakers essentially had to start over each dayas they attempted to decrypt the intercepted mes-sages from that day. To attempt to meet the obvi-ous requirements for both speed and accuracy, theydeveloped a WoE-based mathematical formalism fora part of the code-breaking process. This formalismcalled for quantifying certain clues they found in theintercepted, encrypted messages—in particular theoccurrence of pairs of identical alphanumeric char-acters in different messages—in the form of WoE tomake decisions about the machine settings on thatday. The resulting decisions minimized the amountof time needed to achieve a predefined, acceptablelevel of accuracy.(11,24)

A serious challenge to mapping this kind ofWoE-based decision-making approach to policy de-cisions is that the relevant conditional probabilitiesoften cannot be computed reliably and efficiently.Studies of the brain can lend important insights intohow to overcome this challenge. In particular, un-der some conditions the brain does not appear tocompute conditional probabilities explicitly, but in-stead represents sensory information in a mannerthat allows a useful approximation for the WoE tobe easily generated. For example, many forms of sen-sory information, like motion direction, are repre-sented in a systematically organized manner acrosslocalized regions of the brain. This organization al-lows the evidence associated with opposing hypothe-ses (e.g., Is it moving to the right or left?) to beeasily compared and incorporated into a WoE-likemetric.(12)

Real-world data can be similarly organized tofacilitate such comparisons by, for example, takingadvantage of current computer graphics technolo-gies. An increased emphasis on comparative charts,graphs, and geospatially explicit models (or a com-bination of these) that juxtapose key decision infor-mation would help to approximate a weighted bodyof information for use by environmental decision-makers by highlighting data that are known to be

Page 3: Using Our Brains to Develop Better Policy

376 Linkov et al.

more valuable for making the decision of interestby a particular decisionmaker. We suggest that guid-ance for reports used to make policy decisions shouldrecommend specific graphical depictions that areexperimentally shown to enable easier comparisonand pattern recognition, and should suggest the bestways to present the WoE information to differentaudiences.(13)

2.2. Comparing Lines of Evidence

WoE is often used when evidence comes fromdifferent sources and must somehow be comparedand integrated to form the decision. Neurosciencestudies of sensory cue integration can be used toinform this process. Perceptual judgments are oftenbased on an integration of sensory inputs from differ-ent modalities. For example, to determine the direc-tion of self-motion, or heading, the brain uses a com-bination of sensory cues from two primary sources.One source is the visual system, which interpretsself-motion in terms of patterns of images movingacross the retina. The second source is the vestibularsystem, which is sensitive to the motion itself. Likeother forms of cue integration, the brain uses visualand vestibular information to determine headings inan efficient manner. That is, there is higher percep-tual sensitivity when the visual and vestibular cuesare presented together than when either cue is pre-sented alone, and visual or vestibular cues contributeto the process according to their relative reliabili-ties.(14) This efficient integrative process has been ob-served directly in a monkey brain region that encodesself-motion, where certain motion-sensitive neuronsencode particular combinations of visual and vestibu-lar cues.(15) Comparing the outputs of different sub-sets of these neurons, which are organized withrespect to alternative hypotheses about heading di-rection and intrinsic versus extrinsic motion, canprovide fast and reliable judgments about thesehypotheses.

One current approach in the public policy realmthat is consistent with neural implementations of sen-sory cue integration maintains separate processingstreams for disparate data sources that are later com-pared and combined to inform the decision. The U.S.EPA’s guidance for causal assessment recommendssorting evidence by types, weighing that evidencebased on its logical implications, information source,and other proxies for quality,(16,17) and then com-bining the diverse body of evidence for each causalhypothesis. Likewise, multicriteria decision analy-

sis (MCDA) has been used in many environmentalmanagement applications, including selection of sed-iment management alternatives and restoration plansfor coastal Louisiana and Mississippi.(18−21) MCDAaggregates technical, economic, and social data withexpert judgment consistent with WoE frameworks.Given that WoE is used to generate a quantitativeestimate of the quality of each of a set of lines ofevidence, an MCDA model can be designed thatweighs the extent to which each line should be con-sidered with respect to its quality. In doing so, high-quality information informs the decision more thanlow-quality information.

However, many practical challenges still stand inthe way of successfully combining information fromqualitatively different sources for direct comparison.For example, there is debate regarding how to de-velop conceptual models that select, compress, andcompare huge volumes of environmental data fromdifferent sources. We suggest that organizing data tomake rapid comparisons to distinguish between spe-cific competing hypotheses might help policymakersidentify interrelationships between different sourcesof data and the relative importance of sets of evi-dence, thereby enhancing their usefulness. This or-ganization does not necessarily need to be modeledafter populations of neurons. Rather, the importantprinciple is that the data should be organized withrespect to the hypotheses they are expected to sup-port or oppose and their relative reliabilities. For ex-ample, a causal assessment of kit fox population de-cline used a flow diagram to graphically depict causalpathways leading to that decline. The pathways thatcaused more kit fox death are connected by bolderlines. Coyote predation is the most influential causeof kit fox mortality and therefore coyote controlwould most quickly lead to the desired outcome—greater survival of the endangered fox. Depicting thehypotheses as two major, albeit interrelated, path-ways draws the decisionmaker to the salient informa-tion (Fig. 1).(22,23)

2.3. Reaching a Decision

Understanding how the brain terminates the de-cision process and commits to a course of actionprovides further opportunities for informing policydecisions. Under certain conditions, the brain ap-pears to accumulate evidence from each of a num-ber of lines of evidence by an additive function andthen terminate the process of evidence accumulationonce a prespecified value of WoE is reached (Fig. 2).

Page 4: Using Our Brains to Develop Better Policy

Perspective 377

Fig. 1. The conceptual model indicates the supported hypothe-ses for kit fox declines on the Elk Hills Naval Petroleum Re-serve #1, California. The thickness of arrows indicates relativeweights among the causal connections within each hypothesis thatare described elsewhere in the report. A coyote control programis shown to be the most effective and direct management option.

Consistent with statistical decision theory, in particu-lar the sequential probability ratio test,(11,24) this pro-cess appears to govern the tradeoff between speedand accuracy inherent to many simple perceptualtasks and some nonperceptual tasks involving mem-ory and language. That is, because of the difficulty ofprocessing a noisy or briefly presented stimulus, col-lecting more evidence can improve accuracy but alsorequires more time. In contrast, a fast decision mayrequire less time but is typically also less accurate.Prespecifying the amount of evidence needed toreach a decision both establishes an expected level ofaccuracy and minimizes the amount of time neededto achieve that level of accuracy in the face of uncer-tain evidence.

Fig. 2. WoE in individual neurons of the parietal cortex. Mon-keys performing a simple visual decision task reveal a neural pro-cess akin to an accumulation of WoE—during presentation of sen-sory evidence, spike rates increase over time with faster rates forstronger evidence. The behavioral response is initiated when theaccumulated evidence as rate of firing appears to reach a fixedvalue. Adapted from Gold and Shadlen.(10)

A similar scheme, involving prespecification ofthe overall strength of evidence or time needed tocommit to an action, might help to improve the ef-ficiency of policy decision-making processes, espe-cially those stymied by indecision or under pres-sure to produce results in emergency situations.(25)

Guidance should provide techniques for predefin-ing the level of uncertainty that is acceptable to thedecision process and decisionmaker. Data quality ob-jective guidance is a step in that direction, but moreguidance is necessary, including perhaps defining orestimating in advance the environmental, political,human health and well-being, or legal costs of failingto make a decision.

2.4. The Role of Subjectivity

Subjectivity is inherent in the policy decisionprocess, even when evidence has been collectedusing objective methods. Indeed, subjectivity isinherent to even the simplest decision processes that

Page 5: Using Our Brains to Develop Better Policy

378 Linkov et al.

can be studied in the neuroscience laboratory. Forexample, measurements of the speed and accuracy ofjudgments about the presence or identity of a care-fully controlled sensory stimulus are often stronglyaffected by internally generated factors like the ex-pected probability, magnitude, and nature of positiveand negative outcomes.(26) By explicitly manipulat-ing these factors and measuring the effects on bothbehavior and corresponding brain activity, neurosci-entists are beginning to understand how and wherein the brain objective information, like incoming sen-sory data, is combined with subjective factors, oftengenerated elsewhere in the brain, to guide behavior.The nascent field of neuroeconomics has played acentral role in this pursuit, including studies of howthe relationship between present and future bene-fits, risk perception, emotions, sex of the decision-maker, and different forms of uncertainty affect de-cision making.(27)

Thus, it is not realistic for policymakers to ex-pect to eliminate subjective factors completely fromthe decision process. Attempting to do so wouldsimply promote inaction, resulting in continuingexposure to risks and burgeoning financial costs. In-stead, like the brain processes, effective policy deci-sion making should distinguish between knowledgeabout the physical world and subjective factors as-signed by people.

The current WoE-based decision-making pro-cess for Superfund site remediation considers ninedecision criteria ranging from estimates of contami-nant persistence to stakeholder preferences.(28) How-ever, no method is suggested for separating or identi-fying subjective judgments from objectively obtainedscientific data as they pertain to those criteria, andno formal process is provided for weighing the ninedecision criteria. At the very least, a defined pro-tocol for defining and weighing the evidence, andthen assembling a cohesive recommendation, makesthe decision process more transparent in terms ofsources of subjectivity.(23) However, isolating the in-fluence of subjective factors is only a first step towardmore effective decision making. In addition, meth-ods for standardizing how those factors are weighedare also necessary.(29) This approach is being im-plemented in many current decision methods likeMCDA, and we suggest that policy decisionmakersuse MCDA as a guide for managing subjectivity.Future approaches might also benefit from work inneuroeconomics that is providing a deeper under-standing of the specific biases that subjective factorstypically impose on our decisions.(22)

2.5. Adapting to New Information

The importance of learning from experience isrecognized by scientists and policymakers alike. Inthe brain, neural circuits that compute WoE appearto be calibrated by experience to compute quanti-ties that are appropriate for effective decisions.(30)

This calibration might involve the same kinds oftrial-and-error learning processes thought to shapemany aspects of goal-directed behavior, and it cansubstantially improve the accuracy of future deci-sions based on those inputs. These neural mecha-nisms of learning are similar to the forms of adap-tive policy making. Many environmental programsinclude options for updating knowledge often asso-ciated with on-going resource or adaptive risk man-agement. Adaptive management strategies enforcea management policy, monitor this knowledge up-dating mechanism (e.g., river level in a flood riskmanagement plan), and then update the policy whenfeedback from new information suggests that the sys-tem is not producing the desired result. Similarly,learning in neural circuits occurs by monitoring an er-ror or reward signal, and then changing the behaviorpolicy to increase the likelihood of successfully mini-mizing error or maximizing reward the next time thatdecision must be made. Flood-control and hurricane-protection strategies involving river flow and land-use practices in Louisiana take into account the grad-ual loss of coastal islands and subsidence of marshesand dry landscapes. Other forms of active adaptivemanagement have been applied to such widely rang-ing activities as managing elephants in the KruegerNational Park, South Africa,(31) restoring the Ever-glades, USA,(32) and evaluating expansion of miningoperations.(33)

Adaptive policy making might be enhanced byapplying further lessons from the neuroscience oflearning. For example, reinforcement learning rulescan shape many aspects of goal-directed behaviorand are sufficient to account for at least some formsof experience-dependent calibration of how the WoEis used for decision making in the brain.(34) Thesetrial-and-error learning processes can help to iden-tify the sources of evidence that most reliably predictparticular outcomes.(32) Thus, future decisions can bebased on the most reliable data. A similar processmight be incorporated into policy making to adjustWoE-based metrics to account for new information,either over time in the real world or using simula-tions of realistic scenarios to assess future options.At a minimum, outcome assessments of previous de-cisions should become standard practice.

Page 6: Using Our Brains to Develop Better Policy

Perspective 379

3. CONCLUSION

How the brain makes decisions using imperfectinformation is a central question of modern cogni-tive neuroscience. A host of powerful experimen-tal approaches are being brought to bear on thisquestion, including studies that combine decision-making behavior with measurements of brain activ-ity in humans and other species ranging from leechesto monkeys.(10,35,36) These kinds of studies are valu-able because they can yield insights into not only theoutcome of a deliberative decision process, but alsothe nature of the process itself. These mechanisticinsights could inform policy decision making in twoimportant ways. First, despite its irrationalities andinefficiencies,(37) the brain remains by far the mostflexible and complex decision-making tool availableand therefore may be an appropriate model for struc-turing decision-making processes, similar to otherbiologically inspired solutions to real-world problemsin computation, optics, immunology, and other fields.Second, policy decisions must ultimately depend onhuman judgment and thus will be best served bymethods and tools that complement human abilities.WoE is a common thread linking neural decisionmaking with the often unpredictable world of policydevelopment and decision making. Thus, while neu-roscientists continue to advance our understandingof how the brain computes and uses WoE, scientistsshould help policymakers apply that knowledge toensure that high-quality scientific knowledge informsgovernment decision making.

ACKNOWLEDGMENTS

This work was supported in part by the DredgingOperation Environmental Research (DOER). Per-mission was granted by the USACE Chief of Engi-neers to publish this material. The article has beenreviewed and cleared by the U.S. EPA, but does notnecessarily reflect Agency policies.

REFERENCES

1. Obama, BH. Remarks of President Barack Obama—As Prepared for Delivery at Signing of Stem CellExecutive Order and Scientific Integrity PresidentialMemorandum. Washington, DC, 2009. Available at:http://www.whitehouse.gov/the press office/Remarks-of-the-President-As-Prepared-for-Delivery-Signing-of-Stem-Cell-Executive-Order-and-Scientific-Integrity-Presidential-Memorandum, Accessed on July 8, 2011.

2. World Health Organization. Ecosystems and human well-being: Synthesis. Millennium Ecosystem Assessment,

2005. Available at: http://www.who.int/entity/globalchange/ecosystems/ecosys.pdf, Accessed on February 3, 2011.

3. U.S. Department of Agriculture. Water Quality Infor-mation Center. Available at: http://wqic.nal.usda.gov/nal display/index.php?info center = 7&tax level = 1&taxsubject = 596, Accessed on April 19, 2011.

4. U.S. Fish and Wildlife Service. Managing invasiveplants: Concepts, principles, and practices. Available at:http://www.fws.gov/invasives/staffTrainingModule/index.html,Accessed on February 3, 2011.

5. Reinharz E, Burlington LB. Restoration Planning: GuidanceDocument for Natural Resource Damage Assessment Un-der the Oil Pollution Act of 1990. Silver Spring, MD, 1996.Available at: http://www.darrp.noaa.gov/library/pdf/rpd.pdf,Accessed on February 3, 2011.

6. U.S. Nuclear Regulatory Commission. Handbook of Pa-rameter Estimation for Probabilistic Risk Assessment.NUREG/CR-6823. Washington, DC, 2003. Availableat: http://www.nrc.gov/reading-rm/doc-collections/nuregs/contract/cr6823, Accessed on February 3, 2011.

7. Linkov I, Welle P, Loney D, Tkachuk A, Canis L, Kim J,Bridges T. The use of multi-criteria decision analysis methodsto support weight of evidence evaluation in risk assessment.Risk Analysis, 2011; 31: 1211–1225.

8. Weed D. Weight of evidence: A review of concept and meth-ods. Risk Analysis, 2005; 25: 1545–1557.

9. Linkov I, Loney D, Cormier S, Satterstrom FK, BridgesT. Weight-of-evidence evaluation in environmental manage-ment: Review of qualitative and quantitative approaches. Sci-ence of the Total Environment, 2009; 407(19): 5199–5205.

10. Gold JI, Shadlen MN. The neural basis of decision making.Annual Review of Neuroscience, 2007; 30: 535–574.

11. Gold JI, Shalden MN. Banburismus and the brain: Decodingthe relationship between sensory stimuli, decisions, and re-ward. Neuron, 2002; 36: 299–308.

12. Gold JI, Shadlen MN. Neural computations that underlie de-cisions about sensory stimuli. Trends in Cognitive Sciences,2001; 5: 10–16.

13. Tufte, ER, The Visual Display of Quantitative Information2nd edition. Cheshire, CT: Graphics Press, 2001.

14. Ernst MO, Banks MS. Humans integrate visual and haptic in-formation in a statistically optimal fashion. Nature, 2002; 415:429–433.

15. Gu Y, Angelaki DE, Deangelis GC. Neural correlates of mul-tisensory cue integration in macaque MSTd. Nature Neuro-science, 2008; 11(10): 1201–1210.

16. U.S. Environental Protection Agency. Causal Analy-sis Diagnosis Decision Information System (CADDIS).Washington, DC, 2007. Available at: http://www.epa.gov/caddis, Accessed on February 4, 2011.

17. Cormier SM, Suter GW, Norton SB. Causal characteristics forecoepidemiology. Human and Ecological Risk Assessment,2010; 16(1): 53–73.

18. Linkov I, Moberg E. Multi-Criteria Decision Analysis: Envi-ronmental Applications and Case Studies. Boca Raton, FL:CRC Press, 2011.

19. Linkov I, Satterstrom FK, Kiker G, Batchelor C, BridgesT, Ferguson E. From comparative risk assessment to multi-criteria decision analysis and adaptive management: Recentdevelopments and applications. Environment International,2006; 32: 1072–1093.

20. U.S. Army Corps of Engineers—New Orleans District.Louisiana Coastal Protection and Restoration (LACPR)Draft Final Technical Report. New Orleans, LA, 2009.Available at: http://lacpr.usace.army.mil/default.aspx?p=LACPR Draft Technical Report, Accessed on February 4,2011.

21. Kiker GA, Bridges TS, Kim J. Integrating comparative riskassessment with multi-criteria decision analysis to manage

Page 7: Using Our Brains to Develop Better Policy

380 Linkov et al.

contaminated sediments: An example for the New York/NewJersey Harbor. Human and Ecological Risk Assessment,2008; 14: 495–511.

22. U.S. EPA, Analysis of the Causes of a Decline in the SanJoaquin Kit Fox Population on the Elk Hills, Naval PetroleumReserve #1, California. Washington, DC: U.S. EPA, 2008;EPA/600/R-08/130.

23. Suter GW, Cormier SM. Why and how to combine evidencein environmental assessments: Weighing evidence and build-ing cases. Science of the Total Environment, 2011; 409: 1406–1417.

24. Wald A, Wolfowitz J. Optimum character of the sequentialprobability ratio test. Annals of Mathematical Statistics, 1947;19:326–339

25. Cormier SM. A synopsis of immediate and deliberate en-vironmental assessments. Pp. 21–30 in Linkov I, FergusonE, Magar VS (eds). Real-Time and Deliberative DecisionMaking Application to Emerging Stressors. Dordrecnt, theNetherlands: Springer, 2008.

26. Trommershauser J, Maloney LT, Landy MS. Decision mak-ing, movement planning and statistical decision theory.Trends in Cognitive Sciences, 2008; 12: 291–297.

27. Loewenstein G, Rick S, Cohen JD. Neuroeconomics. AnnualReview of Psychology, 2008; 59: 647–672.

28. U.S. EPA, Office of Solid Waste and Emergency Response.Use of Monitored Natural Attenuation at Superfund, RCRACorrective Action, and Underground Storage Tank Sites.Washington, DC, 1999; Directive 9200.4–17P.

29. U.S. EPA. Integrating Ecological Assessment and Decision-Making at EPA: A Path Forward: Results of a Collo-quium in Response to Science Advisory Board and Na-

tional Research Council Recommendations. Washington,DC: U.S. EPA Risk Assessment Forum, 2010; EPA/100/R-10/004.

30. Law CT, Gold JI. Neural correlates of perceptual learning ina sensory-motor, but not a sensory, cortical area. Nature Neu-roscience, 2008; 11(4): 505–513.

31. Venter F, Naiman RJ, Biggs HC, Pienaar DJ. The evolu-tion of conservation management philosophy: Science, envi-ronmental change and social adjustments in Kruger NationalPark. Ecosystems, 2008; 11(2): 173–192.

32. Kiker G, Rivers-Moore NA, Kiker MK, Linkov I, Ascenario-based gaming system for modeling environmen-tal processes and management decisions. Pp. 151–186 inArapis G, Goncharova N, Baveye P (eds). Environmen-tal Secrutiy and Environmental Management: The Roleof Risk Assessment. Amsterdam, Netherlands: Springer,2006.

33. U.S. EPA. EPA obtains changes to West Virginia coalmine permit to significantly protect water and environment.USEPA News Release, 7/27/2010, 2010.

34. Law CT, Gold JI. Reinforcement learning can account for as-sociative and perceptual learning on a visual-decision task.Nature Neuroscience, 2009; 12: 655–663.

35. Heekeren, HR, Marrett S, Ungerleider LG. The neural sys-tems that mediate human perceptual decision making. NatureReviews Neuroscience, 2008; 9: 467–479.

36. Friesen WO, Kristan WB. Leech locomotion: Swimming,crawling, and decisions. Current Opinion in Neurobiology,2007; 17: 704–711.

37. Kahneman D. Maps of Bounded Rationality. Stockholm,Sweden: Les Prix Nobel, 2002.