1

Integrated Environmental Modeling: A Vision and Roadmap for the Future

Author List: Gerard F. Laniak1, Gabriel Olchin

2, Jonathan Goodall

3, Alexey Voinov

4, Mary

Hill5, Pierre Glynn

6, Gene Whelan

1, Gary Geller

7, Nigel Quinn

8, Michiel Blind

9, Scott

Peckham10

, Sim Reaney11

, Noha Gaber

12, Robert Kennedy

13, Andrew Hughes

14

1US Environmental Protection Agency, Office of Research and Development

2US Environmental Protection Agency, Office of the Science Advisor

3 University of South Carolina, Department of Civil and Environmental Engineering, USA

4 University of Twente, Faculty of Geo-Information Science and Earth Observation (ITC),

Netherlands

5 US Geological Survey, National Research Program

6 US Geological Survey, National Research Program

7 Jet Propulsion Laboratory, California Institute of Technology

8 Berkeley National Laboratory, USA

9 Deltares, Netherlands

10 INSTAAR, University of Colorado – Boulder, USA

11 Durham University, Department of Geography, UK

12 US Environmental Protection Agency, Office of the Administrator

13 US Army Corps of Engineers, Engineer Research and Development Center

14 British Geological Survey, Keyworth, UK

2

Abstract: Integrated environmental modeling (IEM) represents a new paradigm, inspired by modern

environmental problems, decisions, and policies and enabled by transdisciplinary science and

computer capabilities that allow the environment to be considered in a fundamentally different

way. The problems are characterized by the extent of the geo-ecological system involved, the

dynamic and interdependent nature of stressors and their impacts, the diversity of stakeholders,

and the integration of social, economic, and political considerations. IEM provides a science-

based structure to develop and organize relevant knowledge and information and apply it to

explain, explore, and forecast the behavior of environmental systems in response to human and

natural sources of stress. During the past several years a number of workshops were held that

brought IEM practitioners together to share experiences and discuss future needs and directions.

In this paper we present the results of these discussions. First, to provide definition and context

we present a comprehensive view of IEM as a landscape involving four principal elements:

applications, science, technology, and community. This is followed by a vision statement and a

roadmap that includes a set of guiding principles, goals, and activities needed to achieve the

vision. The vision is of a global scale IEM community of practice that leverages modern

technologies to streamline the movement of science-based knowledge from its sources in

research, through its organization into databases and models, to its integration and application for

problem solving purposes. Achieving this vision will require that the global community of IEM

stakeholders transcend social, political and organizational boundaries and pursue greater levels

of collaboration, the first focus of which should be the development of standards for publishing

IEM data and models in forms suitable for automated discovery, access and integration. The

education of the next generation of environmental stakeholders, with a focus on transdisciplinary

research, development, and decision making, is also of primary importance.

1. Introduction

Integrated environmental modeling (IEM) is an emerging discipline inspired by the need to solve

increasingly complex real world problems involving the environment and its relationship to

human systems and activities (technological, social, and economic). The complex and

interrelated nature of real-world problems has lead to a need for higher-order systems thinking

3

and holistic solutions (EPA, 2008b; MEA, 2005; Parker et al., 2002). IEM provides a science-

based structure to develop and organize multidisciplinary knowledge. It provides a means to

apply this knowledge to explain, explore, and forecast environmental system response(s) to

natural and human-induced stressors. By its very nature, it breaks down research silos and brings

scientists from multiple disciplines together with decision makers and other stakeholders to solve

problems for which the social, economic, and environmental considerations are highly

interdependent. This movement toward transdisciplinarity (Tress et al., 2005) and participatory

modeling (Voinov and Bousquet, 2010) fosters increased knowledge and understanding of the

system, reduces the perception of ‗black-box‘ modeling, and increases awareness and detection

of unintended consequences of decisions and policies.

While IEM‘s concepts and early models are now more than thirty years old (Meadows et al.,

1972; Bailey et al., 1985; Cohen, 1986; Mackay, 1991), it is only during recent years that it has

became recognized and appreciated by the greater stakeholder community. National and

international organizations have commissioned studies to determine research directions and

priorities for integrated modeling (ICSU, 2010; NSF1; Blind et al, 2005a,b; EC, 2000,

Schellekens et al, 2011). Senior managers in government, academia, and commercial

organizations are restructuring operations to facilitate integrated and transdisciplinary

approaches (EPA, 2008b). Mid-level managers who realize that no single group has the

comprehensive expertise needed for integrated modeling are actively pursuing inter-organization

collaborations (e.g. ISCMEM2; OpenMI, 2009, Delsman et al, 2009). Researchers and

technology developers, the early adopters of the IEM paradigm, continue a range of research to

develop the science and technologies that enable integrated assessment. Environmental assessors

are utilizing IEM science and technologies to build integrated modeling systems that will address

specific problems at varying scales; examples from the US include the Chesapeake Bay (Linker

et al., 1999) and the San Joaquin River (Quinn and Jacobs, 2006). Finally, policy developers and

decision makers are asking for, and processing, information synthesized from holistic systems-

based modeling approaches; such as those suggested by the US Environmental Protection

1 US National Science Foundation (http://www.nsf.gov/funding/pgm_summ.jsp?pims_id=504707)

2 Interagency Steering Committee for Multi-media Environmental Modeling (http://iemhub.org/topics/ISCMEM)

4

Agency (EPA) (see EPA, 2008b) or practiced in the Murray-Darling Basin of Australia (e.g.

ABARE-BRS, 2010).

The purpose of this paper is to present a comprehensive view of the landscape of integrated

environmental modeling, a vision of its future, and a roadmap for its navigation. The roadmap

consists of goals and activities (destinations and directions) articulated within a comprehensive

contextual framework (landscape). In presenting this comprehensive view of the landscape, we

will attempt to establish a common vantage point with our audience, provide adequate

background, and remain open to feedback from the community. The summaries provided herein

are not intended to be comprehensive, but rather provide an accurate snapshot of the current

state-of-the-art. Further, we do not propose an implementation plan (travel itinerary) – that will

be saved for a later effort.

The primary motivation and input for this paper is drawn from a series of workshops held during

the past several years (Table 1), each of which involved national and/or international

communities. These workshops were convened to share knowledge, experience, and future

visions related to IEM. While the full range of stakeholder perspectives and interests were

represented to the extent possible, the workshops were primarily attended by environmental

modelers and technology developers. Thus, our focus here is the future of IEM in the context of

complex environmental decision making.

The next three introductory sections lay the groundwork for describing the IEM landscape. First,

we present several terms closely related to IEM in an attempt to define them for purposes of this

paper. We then describe the IEM process and its role within the larger decision and policy

contexts. Finally, we organize IEM into four elements: applications, science, technology, and

community. These elements represent layers of the IEM landscape and are described in Sections

2 through 5. Section 6 presents the community vision and roadmap for the future of IEM.

5

Table 1: IEM Workshops

Workshop Title Sponsor Date Organizations Represented Outputs

Environmental Software Systems Compatibility and

Linkage Workshop

US NRC

DOE

March

2000 >40 attendees

A, B, C, D, E Report: NRC (2002)

Integrated Modeling for Integrated Environmental

Decision Making US EPA

January

2007 >100 attendees

B, F, G

Report: EPA (2007)

White Paper: EPA (2008b)

Collaborative Approaches to Integrated Modeling:

Better Integration for Better Decision making US EPA

December

2008 >50 attendees

B, H, I, J, K, L, M, N Report: EPA (2008a)

iEMSs 2010 Conference

Science session: Integrated Modeling Technologies

Workshop: The Future of Science and Technology

of Integrated Modeling

iEMSs July 2010

>75 attendees

International Conference (most

organizations listed below and others)

Workshop Materials

The International Summit on Integrated

Environmental Modeling

BGS

USGS

US EPA

December

2010

>50 attendees

International Conference (most

organizations listed below and others)

Report

AUS NRC: US Nuclear Regulatory Commission (http://www.nrc.gov/) BUS EPA: US Environmental Protection Agency (http://www.epa.gov) CDOE: Department of Energy (US) (http://energy.gov/) DUS ACoE: US Army Corps of Engineers (http://www.usace.army.mil) ENGO: Non-Governmental Organizations FEC: Environment Canada (http://www.ec.gc.ca/) GEU: European Union (http://europa.eu/) HISCMEM: Interagency Steering Committee for Multi-media Environmental Modeling (US Federal Agencies) (http://iemhub.org/topics/ISCMEM) ICEH UK: Center for Ecology and Hydrology, UK (http://www.ceh.ac.uk/) JiEMSs: International Environmental Modeling and Software Society (http://www.iemss.org/society/) KOGC: Open Geospatial Consortium (http://www.opengeospatial.org/) LCUAHSI: Consortium of Universities for the Advancement of Hydrologic Science, Inc. (http://www.cuahsi.org/) MOpenMI: Open Modeling Interface (Association) (http://www.openmi.org/) NUSDA: US Department of Agriculture (http://www.usda.gov)

CSDMS: Community Surface Dynamics Modeling System (http://csdms.colorado.edu/wiki/Main_Page)

NRC (Italy): National Research Council (Italy) (http://www.cnr.it/sitocnr/Englishversion/Englishversion.html)

NSF: National Science Foundation (http://www.nsf.gov/)

ONR: Office of Naval Research (US) (http://www.onr.navy.mil/)

NASA: National Aeronautics and Space Administration (US) (http://www.nasa.gov/)

USGS: US Geological Survey (http://www.usgs.gov)

NOAA: National Oceanic and Atmospheric Administration (US) (http://www.noaa.gov/)

BGS: British Geological Survey (http://www.bgs.ac.uk/)

6

1.1 Common Terms

Within the literature, several terms related to ―integrated environmental modeling‖ are used (see

Box 1). Multiple terms create a distracting confusion for practitioners and users alike. From our

view, the terms are more similar than dissimilar; and are collectively defined within the greater

context of environmental decision making and policy development. In any of the definitions, we

perceive the term integrated to convey a message of holistic or systems thinking (sensu Tress et

al., 2005); assessment as a message of policy relevance (Tol and Vellinga, 1998); while

modeling indicates the development and/or application of computer based models.

Box 1: Terms Related to Integrated Environmental Modeling

Conventional Modeling: a process of creating a simplified representation of reality to

understand it and potentially predict and control its future development. Models can come

in a variety of forms and implementations, including mental, verbal, graphical,

mathematical, logical, physical, etc. (Voinov, 2008).

Integrated Modeling: includes a set of interdependent science-based components (models,

data, and assessment methods) that together form the basis for constructing an appropriate

modeling system (EPA, 2008b; 2009).

Integrated Assessment: seeks to provide relevant information within a decision making

context that brings together a broader set of areas, methods, styles of study, or degrees of

certainty, than would typically characterize a study of the same issue within the bounds of a

single research discipline (Parson, 1995; Weyant et al., 1996; Jakeman and Letcher, 2003).

Integrated Assessment Modeling: an analytical approach that brings together knowledge

from a variety of disciplinary sources to describe the cause-effect relationships by studying

the relevant interactions and cross-linkages (Rotmans and van Asselt, 2001; Rosenberg and

Edmonds, 2005).

Integrated Environmental Decision Making: an approach for evaluating complex

environmental problems holistically by integrating resources and analyses to address the

problems as they occur in the real world; including input from appropriate stakeholders

(EPA, 2000).

Integrated Environmental Strategies (IES): enables researchers to quantify local co-

benefits that could be derived from implementing policy, technology, and infrastructure

measures at a national scale (e.g. reducing air pollutants and GHG emissions) (EPA, 2004).

Participatory Modeling: a generic term used for modeling strategies that rely upon

stakeholder involvement and participation in various forms. In various applications also

known as group model building, mediated modeling, companion modeling, shared vision

planning, participatory simulation, etc. (Voinov and Bousquet, 2010).

7

We also wish to be clear about terms that describe integration of knowledge from more than one

science discipline: multidisciplinarity, interdisciplinarity, and transdisciplinarity. Tress et al.,

(2005) reviewed the literature related to these concepts and suggested definitions in the context

of landscape ecology research; we adapt and apply them here to IEM. These terms differ by the

degree of interaction among disciplines and whether or not non-science oriented stakeholders are

involved. Multidisciplinarity involves participants who essentially execute a portion of a project

related to their individual discipline. Knowledge is exchanged, but not for the purpose of creating

new integrative knowledge or approaches. Interdisciplinarity involves the integration of

knowledge and methods from multiple disciplines in order to form holistic solutions to problems.

Integrating social, economic, and environmental sciences to create a new modeling system to

address complex problems is an example of interdisciplinarity. Transdisciplinarity involves the

integration of knowledge and methods from multiple disciplines (i.e., interdisciplinarity) and the

participation of non-science oriented stakeholders. Transdiciplinarity is focused on societal

issues that require a wide range of science knowledge that is organized and applied via a fully

participatory process. Transdisciplinarity is the highest form of knowledge integration and the

ultimate goal of IEM efforts.

Finally, the term ―stakeholder‖ is generally used to refer to participants in the decision making

process. We group these participants according to their primary role in the overall decision

process. Decision stakeholders are primarily concerned with the problem, its impacts,

representation of interests/concerns, management scenario development, and decisions related to

solving the problem. These stakeholders include those responsible for policies and decisions,

those affected by the problem (including benefits and drawbacks of solutions), including the

public and industry, advocacy groups that support the environment and industry, and assessors

who manage the overall process. Science stakeholders are primarily concerned with the

organization and application of science-based knowledge in the form of data, models, and

methods for the purpose of informing decisions. These stakeholders represent multiple

disciplines and include scientists, modelers, technology developers, and assessors. Assessors

play a unique role as their principal concern is split equally between the decision and the science.

In addition to their technical responsibility, assessors often act as a liaison between the decision

and science stakeholders, facilitating conversations and information exchanges.

8

These two categories of stakeholders are often called experts (Krueger et al., 2012). There are

also stakeholders at large who are the citizens, communities and individuals who may be

impacted directly or indirectly by the decisions made, yet may not be part of the decision making

process. In some cases the boarder line between them and experts can be quite vague because

these stakeholders can also be experts in various domains contributing local knowledge and

experience to the process, which becomes an important factor in participatory modeling (Voinov,

Gaddis, 2008).

In addition to stakeholders involved in specific problems and decisions, we recognize

stakeholders in the broader context of the discipline of IEM. These IEM stakeholders include

those listed above along with the addition of research funding organizations, researchers

representing integration science and individual contributing disciplines, and students. The roles

and interests of stakeholders in this context include:

Advancement of the state of knowledge and practice of complex environmental decision

making and integrated environmental science and modeling.

Prioritization and funding of IEM research and development.

Education of the present and future generations of stakeholders.

1.2 The IEM Process and Decision/Policy Context

Transparency is an important requirement for conducting the decision process. The foundation of

transparency is the articulation of the steps and assumptions of the process and the manner in

which the process is executed. Figure 1 illustrates the overall integrated environmental modeling

and decision process. Similar descriptions are presented by Zagonel, (2002); Jakeman et al.,

(2006); and Liu et al., (2008). The process contains two principal loops: the decision/policy loop

and the modeling/monitoring loop. The process begins with the formulation of a problem

statement that defines the causes of concern, policy or decision context, boundaries and

objectives, management scenarios and options, solution criteria (including tolerance for

uncertainty), and resource constraints. System conceptualization follows and is a high level view

of the social-economic-environmental system within which the problem occurs. It represents the

9

common view of the system shared by all stakeholders and ties the decision loop directly to the

more quantitative modeling/monitoring loop. It forms the basis for developing a detailed

modeling and/or monitoring based solution. This paper focuses on integrated modeling, but

monitoring is also important as both a strategy for responding to problems and as a critical

source of data for modeling. Indeed, data can be treated as models (Voinov, Cerco, 2010) that are

identified and integrated with other models to form a modeling system that is then executed to

simulate cause-and-effect relationships among stressors and impacts. The modeling results are

synthesized to forms useful to decision makers, whereupon the elements of the problem

statement and management scenarios may be modified and the process repeated until satisfactory

policies/decisions are rendered.

Feedback and iteration are essential aspects of the modeling process in general (Jakeman, et al.,

2006), and also the stakeholder-driven IEM and decision process. To execute each step, two

questions must be answered: ―What is needed?‖ and ―What is possible?‖ What is needed is

informed by the previous step(s), while what is possible is informed by knowledge related to

subsequent step(s). The need for iteration and feedback enters because, practically, specifying

what is needed is dependent upon what is possible -- and what is possible is uncertain until it is

attempted. The degree of feedback and iteration is generally a function of the complexity and/or

uniqueness of the problem to be solved. With each repetition of a particular application (i.e.,

combination of problem statement and solution strategy) the process becomes increasingly

prescriptive and sequential.

1.3 Organization of IEM Elements for Discussion

This paper discusses four primary elements of IEM: applications, science, technology, and

community. Applications represent the structured efforts required to address a complex

environmental problem within a decision/policy context (i.e., implementation of the process

shown in Figure 1). The science of IEM focuses on the need to integrate knowledge (data,

models, methods) from multiple disciplines and create a holistic representation and simulation

capability of the relevant social-economic-environmental system. Technology represents the

primary means by which the science of IEM is expressed and applied; including both software

and hardware. Community reflects the social context within which the discipline of IEM evolves.

10

The community plays a vital role in establishing the science domain, organizing the domain

knowledge, identifying and promoting best practices, facilitating professional interactions, and

peer review. Together these elements represent the principal views of the IEM landscape we

wish to present.

Figure 1: The Integrated Environmental Modeling and Decision Process

2. IEM APPLICATIONS

During the past two decades there has been a steady evolution toward IEM in the range and

complexity of environmental issues and problems, related decisions and policies, and the

modeling performed to inform the decisions. While decision makers will continue to address

traditional problem sets involving environmental quality standards and compliance, management

challenges are now framed in ecological, social, and economic terms (MEA, 2005). IEM

applications are the stakeholder community‘s methods for selecting, organizing, integrating, and

processing the combination of environmental, social, and economic information needed to

inform decisions and policies related to the environment. Box 2 lists the characteristics that

11

distinguish IEM applications from more traditional environmental modeling applications.

Combinations of these characteristics manifest in IEM applications to varying degrees. The

sections that follow contain examples to illustrate the range of problems and decisions that

require IEM, and describe future needs and directions as discussed during the workshops (cf.

Table 1).

Box 2. Characteristics of IEM Applications

Problems involve the inter-dependent relationships among multiple stressors, a well-

defined environmental system consisting of multiple compartments (e.g., air, water, soil,

biota, people), and multiple, often competing, endpoints of concern

Decision and policy objectives target resolution of complex environmental problems/issues

in the context of the social and economic landscapes in which they occur.

Solutions are designed to communicate understanding and modeling estimates (predictions)

of environmental system responses to stress and the relative merits of alternative

management strategies.

Decisions and policies involve a diverse set of stakeholders with a high degree of

interaction and information sharing among science and non-science oriented stakeholders.

Solutions require holistic approaches that involve integrating multidisciplinary data,

models, and methods.

2.1 Examples of IEM Applications

The characteristics identified in Box 2 are a suitable way to categorize IEM applications in order

to present an organized view of the IEM landscape. The literature contains a growing number of

studies involving the application of integrated environmental modeling concepts and approaches;

examples of these applications within each category are presented in Table 2.

2.1.1 Future Needs for IEM Applications

Dominant themes throughout workshop discussions (see Table 1) concerning future IEM

applications included adaptive management, stakeholder involvement, best practices, education,

and reusability. Adaptive management refers to the realization that with respect to the

complexities of the human-environmental system we are ―learning as we go‖. We cannot predict

the future with high degrees of certainty. Thus, our decision and policy making process must

accommodate and adapt to new knowledge and data. Given this reality we need to reflect this

adaptability in the design and execution of IEM applications.

12

While the importance and value of involving the full stakeholder community in the decision

process is recognized there remains a significant need for methods and protocols that ensure

appropriate incorporation and integration of perspectives, knowledge, and values.

We also need to document and promote best practices. There is a wide range of approaches that

do not follow any formal protocols, standards, or practices. This makes it very difficult to

understand, review, and reuse applications. There is a need to move toward conceptual

standardization of IEM applications. That is, defining and documenting IEM applications

according to a common process and set of practices. There is a need to establish criteria for

independent peer review and QA/QC, as well as methods for calibration and validation most

appropriate for IEM applications. Best practices should also be organized and made accessible

for purposes of training, education, and interdisciplinary communications.

Reusability of IEM applications implies two things: transferability and interoperability.

Transferability means that IEM applications can be implemented at geographic locations and

time periods other than where and when they originate. Interoperability means that new

applications can be constructed from components used in existing applications. Making this a

reality will require a combination of application metadata (i.e., information that defines the

applications) and technology standards (e.g., software standards for storing and accessing the

metadata and applications). This combination of metadata and standards must be developed and

implemented in direct collaboration with IEM technologists.

3. IEM SCIENCE

The science of integrated environmental modeling provides the knowledge base and integrative

strategies that underpin problem-solving applications. The scientific goals of IEM are to provide

data, models, and methods to explore, explain, and forecast system response to perturbations in

natural or managed environmental systems. To conduct IEM, scientists do not directly pursue

new knowledge within individual disciplines, but rather concern themselves with issues that arise

when domain-specific knowledge bases must be integrated. Box 3 lists the unique characteristics

13

of IEM science. In the following sections we describe these characteristics and present related

future needs and directions.

14

Table 2. Categrories of IEM Applications.

Categories Examples and References

Inter-dependent relationships

among multiple stressors, multiple

environmental compartments, and

multiple endpoints

San Joaquin River Deep Water Ship Channel assessment near Stockton, CA, USA (Jassby and Cloern,

2000; Lehman et al., 2001; Quinn and Jacobs, 2006)

Organic and metal loadings to the Venice Lagoon, Italy (Sommerfreund et al., 2010)

Life-cycle analyses of chemical-based stressors (organics and metals) at a global scale (Sleeswijk and

Heijungs, 2010)

Multi-model comparison of water quality modeling, Pinios River, Greece (Makropoulos et al., 2010)

Groundwater-surface water flooding; linked modelling approach UK basins (Hughes et al, 2010)

See also: Mackay (2001) and Fenner et al., (2005)

Ecological applications focused on

decision/policy objectives with

alternative management strategies

Farber et al. (2006) explore the role of integrated ecological–economic models in quantifying the trade-

offs among ecosystem services in complex, dynamic systems.

Bolte et al. (2006) and Guzy et al. (2008) address important issues for understanding policy effects in

social-ecological systems by agent-based modeling of land use and land cover.

Hulse et al. (2008) alternative future actions in western Oregon‘s Willamette River Basin.

Boumans et al. (2002) describe a unified metamodel of the biosphere developed to simulate the earth

system as comprised of the atmosphere, lithosphere, hydrosphere, biosphere, and anthroposphere.

See also: Maxwell and Costanza (1995); Daniels (1999); Noth et al., (2000); Costanza et al. (2002);

Sengupta and Bennet (2003)

Applications involving a diverse

set of stakeholders Stahl et al. (2011) had science stakeholders work with decision stakeholders to apply a multi-criteria

integrated resource assessment (MIRA) method (Stahl et al., 2002)

Solutions requiring holistic

systems-based approaches that

involve integration of

multidisciplinary data, models, and

methods

Akbar et al. (this issue) establish a monitoring/modeling system capable of informing real-time decisions

related to preparing for and responding to hurricanes impacting the Gulf of Mexico region along the

southeastern United States.

Babendreier and Castleton (2005) describe a national scale assessment of the impacts of chemical

releases from land based waste management units.

15

3.1 System Conceptualization

A system conceptualization captures the essence of the real-world problem within the context of

the processes, cycles, and flows that characterize the relevant social-economic-environmental

system. It establishes a view of the system shared by all stakeholders, thus forming a framework

for understanding and communication. It provides the basis for developing the more detailed

modeling (and perhaps monitoring) methodology and the strategy for synthesizing modeling

results.

The level of complexity of the conceptualization is an important consideration (Robinson, 2006).

It must include a combination of system processes, cycles, and flows that represent decision

control points (i.e., those system elements that can be managed by decision makers) and the state

of the system (e.g., environmental quality, economic impact). The formulation of a conceptual

Box 3: Characteristics of IEM Science

System conceptualization ensures a holistic and transdisciplinary representation of

relationships among the elements of the system; processes, stressors (forcing functions),

impacts, and management options (controls) that are important for mitigation, remediation,

restoration, sustainability, system resilience, etc.

IEM methodology represents the science of integration as applied to environmental modeling:

integration of science knowledge in the form of multiple disciplinary science components

(knowledge, data, models, methods) to form a system simulation capability that is both

coherent and transparent, and transcends disciplinary priorities and perceptions.

Data requirements are diverse and represent multiple science domains. They are produced by

similarly diverse sources whose delivery formats are not typically designed for cross-

disciplinary integration. Data may represent widely varying dimensions related to modeling

(e.g., spatial and temporal scales, population demographics, economic markets).

Model evaluation methods address performance of both individual components and the

integrated system. Performance is defined as a combination of transparency, refutability, and

uncertainty quantification.

Synthesis of modeling results is designed to serve the needs of each stakeholder for

information that contributes to their knowledge of how the system works and defines the

uncertainty related to modeling results.

16

model is a social as well as a scientific process. A primary challenge is translating and merging

the mental models of stakeholders with scientific models for explaining system relationships and

dynamics and making the merged modeling layer transparent, adaptable, and responsive to

queries (Lancaster, 2007; Otto-Banaszak et al., 2011; Voinov and Bousquet, 2010; Voinov et al.,

2008; Zagonel, 2002).

There are several methods and formalisms available for developing and documenting a

conceptual model. Luna-Reyes, 2003 presents examples of mapping tools used to graphically

represent dynamic systems. Fischenich, 2008 describes the purpose, constructs, and relative

advantages and disadvantages of different conceptual models. Other formalisms, based upon

specific quantitative modeling approaches, include systems dynamics (Boumans et al., 2002;

Fenner et al., 2005; Muetzelfeldt and Masheder, 2003), fuzzy cognitive mapping (Özesmi and

Özesmi, 2004), and Bayesian inference (Reckhow, 2003).

3.1.1 Future needs for system conceptualization

Despite the availability of methods and evidence that engaging stakeholders in the system

conceptualization process is a growing priority (Voinov and Bousquet, 2010), the methods

currently employed in practice rarely conform to any specific social or science-based guidelines

or protocols. Looking to the future, there is a need to develop a more structured process

associated with the development and documentation of the conceptual model. In particular, the

following needs should be addressed:

Continued research and documentation of best practices with respect to the social

process of eliciting and merging the array of stakeholder mental models.

Explicit mapping of the elements of a conceptual model to elements of the problem

statement and the quantitative modeling construct.

Documentation of the conceptual model in machine readable metadata to facilitate

stakeholder understanding and communication both within and across applications.

17

3.2 IEM Methodologies

An IEM methodology is a combination of a modeling system and an implementation strategy.

The modeling system represents an integration of data and knowledge from across relevant

science domains and is the quantitative and computational form of the conceptual model. The

mathematical form of the modeling system may be empirical, statistical, process based, or a

combination of the above (Linker et al., 1999; Hart et al., 2009; Schwarz et al., 2006). The

implementation strategy specifies how the modeling system will be deployed in the context of an

IEM application. Deployment may include such strategies as applying the modeling system to

representative locations across a regional or national landscape (Babendreier and Castleton,

2005), executing the modeling system within a monte carlo simulation protocol to address

uncertainty (Johnston et al., 2011), or applying the modeling system repeatedly within an

adaptive management strategy (Akbar et al., this issue). Multiple implementation strategies may

be applied with the same modeling system.

Workshop discussions of IEM methodologies focused on issues related to the construct of IEM

modeling systems, namely; integral vs integrated modeling systems, complexity, and coherence.

Figure 2 illustrates the conceptual difference between integral and integrated constructs. An

integrated modeling system (Figure 2A) uses computer software to link components (e.g., legacy

models) that represent processes or subsystems within the larger system of interest. On the other

hand, integral models represent a singular formalism built from the merger of knowledge of

subsystems or system views held by various disciplinary communities (e.g., legal, economic,

political, social, environmental, Figure 2B). Each of these approaches result in a holistic

modeling based representation of the full system.

The two approaches can also be described as model integration vs. knowledge integration and

the choice between the two has both theoretical and operational implications for IEM (Voinov

and Shugart, 2011). Although examples of each exist in the literature, integrated modeling

systems dominate, because of the IEM community‘s desire to leverage the inventory of legacy

models. It is relatively easier to link existing models than to conduct an intensive participatory

process to merge knowledge bases and build an integral model. However, there are several

18

science integration issues that must be addressed; in the long run, it is not clear how the

resolution to these issues will influence the choice of IEM model system construction.

Figure 2. Two ways of integration. In an integrated model (A) models of components and subsystems are

linked to build a representation of the whole system. In integral models (B) whole system models, based on

disciplinary, or stakeholder, priorities and paradigms, are merged through a participatory process to form a

new and singular formalism representing the system (Kryazhimskiy, personal communication).

Complexity reflects the level of detail at which the problem relevant system is modeled and

includes breadth (how many processes and interactions to include) and depth (dimensional scale

and resolution, e.g., spatial/temporal). We know that complexity should be a direct function of

the problem statement, decision objectives, and the conceptual model; we also know that it is

impossible (and unnecessary) to include all known science related to the social, economic, and

environmental disciplines (Liu et al., 2008). The challenge is to determine which of the detailed

processes are important in simulating the system behavior at an appropriate scale of application

(Sidle, 2006). Oreskes (2003) describes the complexity paradox. As understanding increases,

then the natural reaction of any scientist is to add complexity to their models. In other words, as

data is collected, and understanding correspondingly improves, then more and different

processes could be added to any model. However, as more processes (and subsequent

parameters) are added, the overall uncertainty in the model increases (Hanna, 1988). Further, it is

difficult to know if the tests of increasingly complex models are meaningful (Oreskes, 2003).

19

So a more complex model better captures the nuances of the natural system, but it is more

difficult to determine whether the model successfully reproduces the natural system. More

complex models that emerge from integration are also likely to be more difficult to calibrate and

test (Voinov, Cerco, 2010). This has important implications for complex systems of linked

models.

Another central theme related to IEM methodologies is coherence. Coherence exists when

components are scientifically consistent across the system with respect to complexity, data

requirements, and uncertainty (EPA, 2008b). When designing a modeling system, the following

elements (among others) of coherence must be addressed:

Conservation of energy, momentum, and mass across the modeling system.

Geo-spatial and temporal boundaries of models

Representation and processing of time and space

Differing scales and resolution of dimensional variables between modeling components

(e.g., land use categories, ecological species, human demographics, etc.)

Coherence for integral modeling constructs is addressed at the design stage of the system and

becomes an intrinsic part of the modeling system. When constructing an integrated modeling

system, the issues of coherence must be addressed across the individual components with

appropriate adjustments made at the interfaces.

3.2.1 Future needs for IEM Methodologies

In moving forward, there are several important questions that should be asked concerning our

application of integral and integrated modeling systems. How are the two approaches

complementary? What can we learn from our experience associated with combining various

forms of knowledge (personal experience, group preferences, expert opinion, science-based

knowledge, etc.) in a participatory modeling framework to improve integration between

components? Reciprocally, how can component integration help us better integrate knowledge of

stakeholders? What guidance can we formulate to help make the decision at the problem level

more effective and transparent?

20

With respect to complexity, there is a need to develop better understanding of the relationship

between the elements of a problem statement, conceptual model, and quantitative model. The

degree of complexity should match the need. This is easy to state but difficult to implement,

especially when models are assembled from components previously developed to serve very

different needs. The community also needs guidelines and protocols for constructing

appropriately complex modeling systems for IEM. Similarly, guidelines and tools for achieving

coherence are needed, particularly in cases where information is shared between components of

different science disciplines (e.g., economic and environmental) and among legacy models that

are integrated to form larger modeling systems.

3.3 Data for IEM

Three important uses of data within the context of modeling are: data used in model development

(parameter estimation); data as input to models; and data for evaluating models. Table 3 lists the

range of environmental data relevant to IEM and Figure 4 illustrates the geo-spatial nature of

much of the data. The data draw from several science disciplines (multidisciplinarity) and span

most spatial and temporal scales. Data preparation for IEM is a resource intensive task; the steps

of the process are described in Box 4.

Table 3. Categories of IEM Data

Environmental Features, Characteristics, and

Observations

Atmosphere

Meteorology

Watersheds

Surface Soils

Land use / Land cover

Digital Elevations

Vadose zone

Groundwater systems

Surface water systems

Ecosystems

Sources

Waste management units, waste properties

Agricultural settings and practices

Industrial settings and practices

Natural events

Geographic/Topographic Boundaries

Administrative

Political

Ecoregions

Watersheds/Waterbodies

Aquifers

General Stressors

Chemical properties

Land use change scenarios

Global warming scenarios (future precipitation

and temperature)

Natural events

Exposure Pathways and Health Effects Factors Population Distributions, Demographics, and Behaviors

21

Human

Ecological

Human

Ecological (aquatic, terrestrial)

Figure 4. Example geo-spatial data layers for IEM. Image courtesy of RTI, International.

Box 4: Steps for Data Preparation

1) Discovering data sources: that may be located on the internet, in a journal article or

government report, or in a private data file,

2) Devising methods for accessing the data which requires understanding the semantics

and storage formats of each source and devising extraction methods,

3) Processing the data which includes performing mathematical/statistical transformations

and QA/QC of the data to ready it for the application,

4) Integrating the data which establishes system coherence and connectivity, and

5) Building input files which requires placing the data into individual model input files.

Executing these steps requires reconciliation of varying semantics across the numerous data

sources (Box 4, steps 2 and 3) and establishing an operational ontology that specifies, for

models, the relationship among physical entities of the system (Box 4, steps 4 and 5). The

ontological relationships include the geo-spatial distribution of environmental features (Figure 4)

22

and the connectivity of these features that define movements and flows through the system (e.g.,

individual catchments connectivity to individual stream segments to specify water runoff routes).

Currently, the execution of this process for individual applications can be characterized as a

semi-manual task of cutting and pasting file fragments to form a single coherent dataset. Needed

data are dispersed among a myriad of sources, each representing data with its own unique

combination of vocabulary, storage formats, and accompanying descriptive information

(metadata). Significant effort and science-based judgment are required to locate, access, process,

and prepare data for IEM applications. Complicating the issue is the fact that new sources of data

are coming online seemingly every day, e.g., real-time remote and in situ sensing, social media.

3.3.1 Future needs for IEM data

The resource intensive task of gathering and organizing data for IEM applications represents a

barrier to the wider acceptance and use of IEM to support decisions and policies. The challenge

for IEM is discovering and accessing the numerous sources of data and building an integrated

dataset that provides a single coherent representation of the system to be shared by all modeling

components. Because major portions of the IEM community (e.g., nationally) access the same

primary sources of data and process it to the same end (i.e., model inputs) we can gain

tremendous efficiencies if we view this challenge as a community infrastructure need. In short,

there is a need to develop community accessible tools and methods for: (a) automated discovery,

access, and retrieval of data from individual sources; and (b) ensuring quality and facilitating

integration of data from multiple sources and populating model input files. Proper documentation

(metadata) and standards for storage, retrieval and update are of paramount importance for this

end.

The solution to this science-based need is dependent on technology and thus, Section 4 describes

ongoing and emerging efforts to improve access to environmental data and suggest future needs

and directions as discussed at the IEM workshops.

23

3.4 Model Evaluation

Model evaluation combines quantitative and qualitative information about a modeling system‘s

appropriateness for the problem and ability to characterize the uncertainty of model predictions.

This characterization must be organized for use by all stakeholders. Key attributes of model

evaluation are transparency, refutability, and uncertainty quantification. Together, they establish

the scientific veracity and stakeholder confidence/acceptance of an application and the

information it produces. Transparency requires that all aspects of the application design and

execution be accessible to facilitate understanding and reproducibility. Refutability requires a

hypothesis-testing framework in which data are used in specific ways to test the model‘s ability

to simulate the system of interest. This may include meaningful model validation efforts, in

which the model is tested with data not included during its development. This may require the

involvement of decision stakeholders because the ultimate test of a model is always its utility and

usability by end users. Refutability is difficult for any model of an environmental system and

evaluating an IEM system is even more challenging.

Challenges related to prediction and uncertainty in integrated modeling were experienced in

recent climate change modeling (e.g. IPCC, 2007). Several textbooks on these subjects have

been produced in recent years, and an active scientific community continues to explore new

methods. A few references include: Menke (1989) and Aster et al. (2005) from geophysics;

hydrology (Beven, 2009; Clark et al., 2011; Hill and Teideman, 2007); and Saltelli et al. (2008)

from econometrics. However, application of these ideas to more complicated IEMs is only

beginning to emerge, (Beven, 2007; Refsgaard et al, 2007; Ascough et al, 2008; McIntosh et al.,

2011; Benett et al., 2012) and there is still much to learn in developing methods and case studies.

We expect that the growing use of IEMs will test the limits of existing methods and lead to

additional innovations. In Table 4 we describe basic model evaluation methods that are useful in

a wide range of circumstances and that have characteristics advantageous to IEMs.

24

Table 4. Summary of methods used for model evaluation.

Technique Description

Error evaluation and

propagation

This starts with data being used to construct the component and IEM models, proceeding to the evaluation of predictions. This is critical to IEMs

because data availability and errors in one part of the system invariably affect uncertainties in other parts.

Such holistic considerations of error and uncertainty is critical to integrated model development.

Error-based weighting This is critical to integrated use of data in IEMs and is complicated by the need to synchronize the weights among various disciplines since

variations in the importance of the processes and data may occur. With error-based weighting, weighted discrepancies between observed and

simulated values become comparable, partly because they become dimensionless and partly because error-based weighting puts the discrepancies in

a consistent statistical framework

Sensitivity analysis This can be conducted by combining computationally efficient local linear methods and computationally expensive global methods. Linear methods

are attractive for IEMs because they require on the order of 10s of model runs to obtain useful results

Alternative models These are important in IEM model development for assessing the effects of conceptual model uncertainty. Alternative conceptualizations

considered often affect more than one aspect of the system, and integrated models allow the consequences to be represented realistically throughout

all relevant aspects of the system simulation (e.g., the hydrologic cycle, Clark et al., 2011).

Optimized calibration

methods

These are used to improve objectivity and reproducibility. The optimization process can identify the information content of a given set of

observations and, along with sensitivity analysis methods, can identify important new data. For IEMs, this can be particularly challenging because

of the number of linked/coupled component models involved.

Uncertainty

Quantification

Should be done with the best possible methods, considering the computational demands of the model and requirements of end users of model

results. IEM models require both knowledge and uncertainty from different system components to be conveyed to uncertainty quantification.

Model Tests Should be undertaken against alternative data sets. Often called model validation ―test‖ is used here because the model may be deficient, in which

case the ―test‖ may not validate the model. In IEM models it is important to test component models and integration requires additional tests

throughout the IEM.

Post-Audits Comparing model predictions to what actually happens are called post-audits, and may involve response to changes that take moments, days, years,

decades, centuries, or longer. Post-audits require monitoring. Either errors in economic and/or social evaluations or differences in expected and

actual changes, and in model simulated processes can be highlighted. IEMs provide important opportunities for post-audits because simulated

results can affect resources important to large ecological systems and to many people.

Calibration and testing in

data scarce conditions

Because of complexity, the likelihood of having complete datasets for extensive model analysis is low. Therefore, confidence building and

uncertainty analysis has to be performed in data-scarce conditions. The model and theories used in modeling can suggest specific monitoring foci to

collect the most important data sets to help decrease uncertainty and facilitate adaptive management strategies. However, this means longer delivery

times. Involving stakeholders and other users in model analysis is also essential to judge modeling success.

25

3.4.1 Future needs for model evaluation

The expanding applications of IEM require critical improvements to our model evaluation

methods. The transdisciplinary nature of this modeling requires methods and tools to harmonize

and integrate model evaluation information from different disciplines into a single coherent

representation. Further, there is a need to establish and consistently apply best practices,

including identification of sources of uncertainties and tracking their propagation through a

modeling system.

3.5 Synthesis of modeling results

Like system conceptualization, the synthesis of modeling results represents a key interface

between science and decision stakeholders (see Figure 1). The objective is to interpret,

consolidate, and present the results of complex integrated modeling to stakeholders and decision

makers. Synthesis must produce information that is not only of high scientific quality but, also

useful to the decision makers. McNie (2006) discusses the challenges of ―reconciling the supply

and demand‖ of scientific information between scientists and decision makers, defining ―useful‖

as a combination of salient, credible, and legitimate information. Salience implies contextual

relevance, credibility refers to scientific veracity, and legitimacy refers to a lack of bias.

Decision support systems assimilate the synthesized modeling results and prepare them, along

with other relevant information, for presentation to the decision stakeholders (McIntosh et al.,

2011). While scientists routinely synthesize results for other scientists via scientific journal

articles, the synthesis of scientific information for decision making is not well understood and

executed (NRC, 2005). With IEM, the challenge again is to serve diverse users and stakeholders

from various disciplinary backgrounds and with different vocabularies, priorities and vested

interests. This may require multiple synthesization streams tailored to particular audiences and

designed hierarchically to move seamlessly from very general displays of overall results to

highly detailed, component-based visualizations (Ellarby and Kite, 2006; Liu et al., 2008).

3.5.1 Future needs for synthesization of modeling results

Looking to the future it will be necessary for stakeholders (representing both science and

decision makers) to work more closely together to develop means for synthesizing modeling

26

results for their effective use in decision making. Studies, such as reported in this issue

(Arciniegas et al., 2012), must be encouraged and expanded. The participatory modeling

approach offers another promising interface with stakeholders, however specific applications that

can make IEMs available for stakeholder use and acceptance are still largely missing (Voinov et

al., 2008).

4. IEM TECHNOLOGY

IEM‘s technology component covers the computational and informatics tools, approaches, and

concepts required for achieving overall IEM objectives. The objectives of IEM are ambitious and

require sophisticated, state-of-the-art technologies to understand integrated environmental

systems (i.e., databases; models; assessment methods; and visualization tools) that span multiple

scientific disciplines, but need to be summarized concisely to support policy and decision

making goals. Box 5 lists the characteristics that distinguish IEM technologies from those

employed in conventional modeling.

Box 5: Characteristics of IEM Technology

Facilitates linkage of and information transfer among components of integrated modeling

systems.

Facilitates data collection from disparate sources, data processing and integration, and

preparation for model consumption.

Manages the execution of integrated modeling systems across software and hardware

platforms.

Provides user interfaces to facilitate information exchange (e.g., scenario development, data

input, data analysis/visualization, and stakeholder involvement)

The technological characteristics associated with IEM are broad and require a diverse set of

informatics and computing approaches. For practical reasons, we limit the scope of this

discussion to two key topics: 1) modeling infrastructures (aka frameworks) and interoperability,

and 2) data access and processing. We discuss each of these in the following sections.

4.1 Modeling Infrastructures and Interoperability for IEM

27

Modeling infrastructure is software designed to facilitate the information and data flow among

components of a modeling system. Components of conventional environmental modeling

systems include user interfaces, data analysis and visualization tools (including GIS), individual

science models, calibration and optimization tools, and perhaps distributed computing tools. In

addition to these, and unique to IEM, is infrastructure needed to construct modeling systems

comprised of multiple science-based models not originally designed to "talk" to each other, i.e.,

they are not designed for interoperability. Interoperability is the ability of different information

technology components, systems, and software applications to communicate and exchange data

accurately, effectively, and consistently, and to use the information that has been exchanged

(IEEE, 1989; Heubusch, 2006). Linking models not designed for interoperation is often

surprisingly difficult, and a brute-force approach to the problem requires a significant investment

of time and effort. The following paragraphs discuss some of the technological difficulties that

arise when tackling model interoperability.

Programming Language Incompatibilities: Models are written in different programming

languages and conversion to a single language is impractical, time-consuming and error-prone.

Languages do not offer the same data types and features, so special tools are required to make

function calls across the "language barrier". If models are not converted to a common language,

then a language interoperability tool must be provided to allow components to share data.

Inadequate Model Documentation and Metadata If models are to be coupled so they can

exchange information during a simulation, it is beneficial for model documentation to be both

human-readable and machine-readable. A detailed, standardized description of each component

should be provided by the scientists and implemented, perhaps as a standard XML file that is

machine readable, but also easily rendered in various human-readable formats (HTML, PDF,

etc.). Documentation and metadata move models from black boxes to being reusable components

within a larger software system. This documentation will ensure that a component is applied

properly.

Spatial and Temporal Incompatibilities Models may have different dimensionality (1D, 2D, or

3D) and generally consist of one or more arrays (1D, 2D, or 3D) that are being advanced in time

28

according to differential equations or other rules. The models generally have a geographic

context and are often used in conjunction with GIS tools. They may use different types of grids

(e.g., rectangles, triangles, and Voronoi cells). Each model has its own time loop or clock, but

when linked they must use a single clock. The numerical scheme may be either explicit or

implicit, time steps fixed or adaptive. Creating a model infrastructure that can operate over a

variety of spatial and temporal scales is challenging, and adding logic that informs modelers

when model components are or are not compatible due to spatial and temporal mismatches is

even more challenging.

Non-standardized Input and Output File Formats One of the most basic forms of incompatibility

between models is inconsistent input and output files. In the atmospheric modeling community,

NetCDF is the de facto file type used, and there are standardized conventions (e.g., CF

conventions) for semantics used within NetCDF files. No such standards exist in environmental

modeling. While there have been attempts to standardize file formats (e.g., the ArcHydro data

model), it has proven difficult to create a single description of the natural environment that can

form the foundation for creating standardized input and output files.

Reliability and Sustainability of Models A key question in reliability and sustainability of models

is component testing--specifically, what type of unit testing should be performed and how often?

As a link in a computational chain, each component must be reliable. Special measures may be

needed to ensure numerical stability, accuracy, and consistency across the full range of intended

use. All modern software builds on existing software, but such dependence must be carefully

scrutinized for future availability. Besides these challenges, there are others related to managing

the software contributions to the system. When software is contributed by a group of developers,

there must be quality assurance checks in place to verify reliability of the software. There must

also be higher level structure to the software contributions so that they do not, for example,

reproduce functionality already within the system.

Classifying Models Wrapping model components with a single, common interface like OpenMI

(Moore and Tindall, 2005) allows them to be linked, but does not address which components are

interchangeable. This requires defining component categories or types and requiring that all

29

components of the same type return some common set of output variables. As long as they

provide this common set, they are permitted to use different input variables and additional output

variables. One advantage of the ―common set of outputs‖ approach is that users no longer need

to connect input and output variables one by one, manually. Instead, they simply connect

components and that connection accesses all of a component‘s output variables.

User-Friendliness Lastly, and perhaps most importantly, the graphical user interface design for

the system must be intuitive for end users, ranging from experienced modelers to the wider

community of stakeholders. Components are more likely to be reused if they provide a user-

friendly GUI for preparing the various types of input data that they require.

Several modeling frameworks that address these issues in a holistic and integrated manner have

been developed during the past decade. Castronova and Goodall (2010) and Lloyd et al. (2011)

summarize a variety of these IEM frameworks that use both procedural and object-oriented

programming languages. Examples include ESMF (Hill et al., 2004) and CSDMS (Peckham,

2010); OpenMI (Moore and Tindall, 2005); SEAMLESS (Ittersum et al., 2008), and OMS

(Ahuja et al., 2005; David et al., 2002). Each of these frameworks employ a set of standards to

address the interoperability issues described above. Unfortunately, the standards are ―local‖, that

is, they apply within the particular infrastructure or framework only. The resulting reality is that

the issues of interoperability described above must be resolved for every combination of science

component and modeling infrastructure. And, more importantly, the sharing of science

components among frameworks, a priority for IEM applications and science, remains difficult

and resource limited. For these reasons, it is common to see IEM modeling systems and

frameworks that contain only a limited number of components; typically that set familiar to the

developers and needed to solve a particular problem.

This issue has been recognized and progress toward increased interoperability is occurring.

Groups have ―discovered by doing‖ that the issues described above are common across

integrated modeling systems and that the required functionality can be abstracted and

standardized. In particular, groups have found that it is necessary to provide separate component

interface functions for initializing a model, stepping through time and updating state variables,

30

finalizing a model, retrieving data from another component, and assigning data to another

component. The simple but critical reason for this is that every model was originally designed to

have its own clock or time loop, but in a coupled configuration only one component can be in

charge of the clock. Components that provide the interface functions listed above are able to

provide their caller with fine-grained control of their functionality, which is key to usability in a

plug-and-play framework. This interface in no way prevents them from being used as stand-

alone models. As an example, the Common Component Architecture (CCA) is a set of

component and framework standards developed within high-performance, scientific computing.

CCA-compliant components can be reused in other CCA-compliant frameworks (e.g. Ccaffeine,

XCAT and SciRUN).

Other examples of groups creating solutions with increased emphasis on interoperability issues

include CUASHI and OpenMI. CUAHSI (Maidment et al, 2009) has designed and implemented

standards for exchanging hydrologic data over the web including the WaterOneFlow web

services and the Water Markup Language (WaterML). These were used to construct a

Hydrologic Information System with a service-oriented architecture and able to integrate

hydrologic observational data from international, federal, state, local, and academic data

providers. The design principle is to keep databases separate and autonomous while providing

standards that allow software programs to query and extract data from them using standardized

approaches. And, the OpenMI Association has designed standards for linking water resource

models at run time that require input or output from other models. When using the OpenMI

paradigm, models are treated as components of a larger simulation system so they must adopt

standard software interfaces in order to be integrated. The OpenMI standard has received

international attention and movement of existing frameworks to accommodate the standard is

occurring.

4.1.1 Future needs for model infrastructure and interoperability

From workshop discussions the highest priority for IEM infrastructure technology is to further

enable interoperability. Communication technologies are central for achieving this goal. They

emphasize protocols for coupling system components using well-documented data models that

can be expressed as object-oriented data structures, database schemas, and mark-up languages,

31

while providing a consistent view of data across systems. Service-orientation is critical for the

system to allow modularity of models that remain interoperable within a larger system. This

means standards. To guide this movement toward higher degrees of interoperability we look to

Wang et al. (2009) who describe six levels of interoperability (Figure 3); in order of increasing

capacity for interoperation: technical, syntactic, semantic, pragmatic, dynamic, and conceptual.

These levels are further characterized as advancing from integratability, to interoperability, to

composability. Differences between these levels reflect the type of information to be exchanged,

not the technology that implements the exchange.

Figure 3. Levels of Conceptual Interoperability Model (Image adapted from Tolk et al., 2006).

A reasonable goal for IEM in the near term is pragmatic interoperability, which according to

Wang et al. (2009), populates a modeling system design with components made available by

different developers. To achieve this with technology, the components must be published with

syntactic, semantic, and contextual information that enables integration choices and decisions.

This information should be in machine readable format. Before technology can automate the

32

interoperability process, scientists and modelers representing the disciplines of IEM must

convene and develop the required linguistics.

In the longer term, the IEM community should focus attention on achieving automated

composability, which makes it possible to construct IEM systems in a truly plug-n-play

environment. Achieving this level of interoperability will require the codification of metadata

representing a combination of problem statement, decision objectives, and conceptual model

developed by the IEM stakeholders. This information forms criteria input to a software system

capable of discovering and scanning web-based repositories and retrieving modeling components

that satisfy the criteria. These components can then be automatically integrated to form an

operational modeling system. In addition, data needed by the newly configured modeling system

will be discovered, retrieved, and processed into input files. This level of interoperation requires

codification of the intelligence and reasoning that formulates the problem statement and creates

individual components.

Perhaps the most interesting aspect of this seemingly daunting challenge is the fact that there is

no new information required to achieve these goals. Each and every IEM application currently

processes all of this information. The problem is that we have no commonly accepted and

applied means of expressing this information, much less rendering it into a machine-consumable

format.

4.2 Data access and processing for IEM

There is growing recognition by governments and academic institutions that access to data that

describes environmental and ecosystem states is critical to solving complex environmental

problems. Table 5 summarizes several ongoing efforts to provide such data at national and

international scales, including government, education, and technical organizations.

As seamless access to data becomes available (Box 4, steps 1 and 2) the next challenge is to

process and transform the data for use in IEM systems (Box 4, steps 3 and 4). This requires

extensive geo-processing, statistical treatment of data, and methods for integrating

multidisciplinary data. Tools and software systems to automate these functions in an integrated

33

fashion are beginning to emerge. The Geoscience network (GEON, Ludascher et al, 2003,

http://www.geongrid.org/) is an open collaborative project funded by the US National Science

Foundation to develop cyberinfrastructure with the ultimate goal of linking heterogeneous

scientific data and information for the purposes of knowledge discovery, sharing, and integration

into scientific workflows. The GEON Open Earth Framework (OEF) is a visual analytics suite

of software libraries and applications that assist in these data processing functions. The suite's

collection of interactive visual techniques are designed to help scientists derive insight from

complex, ambiguous, and often conflicting Earth science data. Also, Data for Environmental

Modeling (D4EM - Johnston et al., 2011) is an open source software system developed expressly

to access, retrieve, and process data for IEM. D4EM is currently linked to several US

Government databases and performs all data processing required to serve data directly to an

integrated modeling system designed to simulate interactions among watersheds and aquatic

ecosystems

4.2.1 Future needs for IEM data and data processing

There are several areas where access and processing of data can be improved. It will be

necessary for the IEM community to increase involvement in national and international

initiatives and to encourage the collection of additional types of data to enable the wide range of

IEM applications and research. Developing more standardized tools to enable discovery of and

access to data for IEM will enhance the community‘s ability to conduct IEM. Developing

methods and accessible tools for processing data into forms needed by IEM will reduce the

necessary effort and increase overall consistency among applications. It will be advantageous to

represent model results as data for direct comparison with observations. This will bring the

information flow full circle and provide researchers, assessors, and decision makers with a

coherent and continuous flow of relevant information.

34

Table 5. Examples of data access initiatives of relevance to IEM

Initiative Geoscope Description

Global Earth Observation System

of Systems (GEOSS)1

Global An international effort to coordinate a comprehensive monitoring of the state of the earth, study large

scale processes, and predict behavior of the earth system

Infrastructure for Spatial

Information in the European

Community (INSPIRE)2

EU The stated purpose is to assist in environmental policy making across national boundaries. The spatial

information considered includes seventeen topical and technical themes originating from numerous

sources throughout the EU

European Shared Environmental

Information System (SEIS)3

EU A web based system where public information providers share environmental data and information. In

implementing SEIS, the EEA is building on existing reporting systems and tools: the INSPIRE directive,

Global Monitoring for Environment and Security (GMES), and (GEOSS.)

Water Information Service for

Europe (WISE)4

EU Gateway to information on European water issues. It comprises a wide range of of data and information

collected by EU institutions to serve several stakeholders.

Environmental Resources

Information Network (ERIN)5

Australia The objective is to organize environmental information from many sources and provide standards based

tools for discovery, access, and use. The information includes maps, species distributions, documents and

satellite imagery, and covers environmental themes ranging from endangered species to drought and

pollution

Environmental Dataset Gateway

(EDG)6

US A gateway developed by the US EPA to web-based information and services. It enables data consumers to

discover, view and access data sets, as well as geospatial tools. Users also have the ability to catalog and

maintain their geospatial metadata contributions via the EPA Metadata Editor Tool

National Ecological Observation

Network (NEON)7

US Collection of data across the United States on the impacts of climate change, land use change and invasive

species on natural resources and biodiversity. Designed to detect and enable forecasting of ecological

change at continental scales over multiple decades.

US based Consortium of

Universities for the Advancement

of Hydrologic Science, Inc.

(CUASHI)8

US Developing the Hydrologic Information System (HIS), an internet-based system that provides for sharing

hydrologic time series data contributed by a wide range of providers, including the National Water

Information System of the US Geological Survey.

35

Open Geo-spatial Consortium

(OGC)9

Global Efforts to work toward global standards for access, retrieval, and use of geo-spatical data.

1 http://www.earthobservations.org/index.shtml

2 http://inspire.jrc.ec.europa.eu/index.cfm/pageid/48

3 http://ec.europa.eu/environment/seis/

4 http://water.europa.eu/

5 http://www.environment.gov.au/erin/about.html

6 https://iaspub.epa.gov/sor_internet/registry/edgreg/home/overview.do & https://edg.epa.gov/EME/

7 http://neoninc.org

8 http://his.cuahsi.org

9http://www.opengeospatial.org/

36

5. IEM COMMUNITY

In these formative years of IEM, the need for structured and unstructured communication, has

been critical. Resolving differences among disciplines and stakeholders is an intensely social

process. Facilitating and documenting social interactions is the foundation of an integrative

process. The IEM community, as represented at the workshops (Table 1), has called for even

greater efforts in this area. Integrated environmental modeling is not just an activity that

discovers, organizes, and delivers knowledge and technology for others to assimilate and apply.

The IEM enterprise (science, technology, community, and applications) is driven by the needs of

others: it exists because problems and decisions require it. Thus, the formation, structure, and

social processes of the broad IEM community are an important area of focus and study. In many

fields and sectors, openness, collaboration, sharing, and social learning have been shown to drive

innovation and growth (Tapscott and Williams, 2006). Structured community processes can

reduce duplication of efforts and increase leveraging of resources and overall efficiency. These

behavioral attitudes and characteristics are often expressed through formal communities of

practice. Lave and Wagner (1991) introduced the term ―community of practice‖ and defined it as

―a set of relations among persons, activity, and world, over time and in relation with other

tangential and overlapping communities of practice.‖ Box 5 lists the key characteristics of an

IEM community of practice.

37

Box 5: Characteristics of an IEM Community of Practice

The community is transdisciplinary, comprised of a diverse set of stakeholders representing

environmental policy and decision making, funding organizations (research and application),

social, economic, and environmental sciences, industry, environmental advocacy groups, and the

public.

Membership is voluntary and participation is open and equally available to all regardless of

organizational affiliation and stakeholder identity.

Interactions involve interdisciplinary communication, sharing, collaboration, and negotiation

oriented toward achieving shared goals of integration.

The community establishes relationships with other communities that may provide specific

discipline or stakeholder components.

Transparency, efficiency, and openness are promoted. Collaboration is encouraged across

organizational boundaries.

5.1 Current state of the IEM community of practice

The early phases of IEM development were conducted by disparate small groups in government,

academia and the private sector working on science and applications, mostly independent of one

another. Results of these efforts were disseminated through traditional outlets such as conference

presentations, technical reports, journal articles, and websites. These groups advanced the

science and application of IEM, but their efforts and products typically were not coordinated and

compatible. These early efforts produced a large number of IEM approaches and functionally

similar and operationally unique software components and systems. For example, there are now

dozens of modeling frameworks, each claiming unique merits (van Evert et al., 2005). Recently,

several larger groups relevant to IEM (not necessarily formal communities of practice)

specifically related to modeling domains or select entities (Table 5) have formed. The emergence

of these communities demonstrates a widely shared belief that topics related to IEM benefit

greatly from the establishment of formal relationships among stakeholders.

38

Table 5. IEM Relevant Communities or Communities of Practice.

Name Scope

ISCMEM1 Formal effort of modeling groups at nine U.S. federal agencies to share ideas, models, and projects.

CSDMS2 Convenes experts to facilitate development and dissemination of integrated software modules that

simulate dynamics of the earth's surface, focusing on the interface between lithosphere, hydrosphere,

cryosphere, and atmosphere

ESMF3

Produces shareable software for climate, weather, and related applications by building high-

performance, flexible infrastructure that increases ease of use, performance portability,

interoperability, and reuse in climate, numerical weather prediction, data assimilation, and other earth

science applications.

OpenMI4

Community of organizations that has proposed a standard for exchanging data between environmental

models.

CUASHI5 Community that facilitates discovery and access to hydrologic data (Hydrologic Information System,

HIS), sharing of hydrologic models and codes (Community Hydrologic Modeling Platform, CHyMP),

and a web portal for interactive access to widely used simulation codes and high performance

computing (HydroHub).

CIEM6 Worldwide open community whose goals are to:

Enhance IEM learning and education (establish best practices; produce and share educational

tools);

Leverage IEM solutions (make them accessible and reusable);

Promote transparent, efficient, and open science and technology;

Facilitate scientific collaboration; and

Allow efficient use of resources (limit duplication in technology development).

1ISCMEM: Interagency Steering Committee for Multi-media Environmental Modeling (http://iemhub.org/topics/ISCMEM) 2CSDMS: Community Surface Dynamics Modeling System (http://csdms.colorado.edu/wiki/Main_Page) 3ESMF: Earth System Modeling Framework (http://www.earthsystemmodeling.org/index.shtml) 4CUAHSI: Consortium of Universities for the Advancement of Hydrologic Science, Inc. (http://www.cuahsi.org/) 5OpenMI: Open Modeling Interface (Association) (http://www.openmi.org/) 6CIEM: Community of Practice for Integrated Environmental Modeling (http://www.iemhub.org)

5.1.1 Future needs of IEM Community of Practice

With the formation of several IEM-related communities and the launch of iemHUB with its web-

based collaboration tools, the community approach is becoming a reality. The challenge is to

foster and promote participation from the full community; which is often easier said than done

(Voinov et al. 2010). This section includes several goals related to establishing a highly

interactive community for IEM: extending the community, removing organizational and

institutional barriers, and formalizing the community of practice.

39

At the present, the IEM community is represented by the individuals who have participated in the

workshops. In order to achieve the goals and conduct the activities proposed in this paper, the

membership of a formal community of practice must be extended to include additional science

stakeholders (i.e., modelers from the disciplines of economics and social systems) and decision

stakeholders (i.e., lay public, citizen scientists, and decision/policy experts). IEM will not move

forward without the ongoing cooperation and collaboration from a comprehensive community.

This is exemplified best by considering the potential contribution of citizen science networks.

Citizen science networks can contribute in many different ways. First, they can directly monitor

natural resources and environmental conditions by gathering information and data that might be

used in IEM applications. Citizen scientists can help engage the broader community in the design

and review of IEM applications by asking the questions that need to be answered, and by

providing greater transparency on the methods by which those questions will be answered. If

interested and enlightened citizen scientists do not understand the essentials of an IEM

application, there is little chance that the broader community and policy/decision makers will.

Citizen scientists can help to educate the general public about IEM science and its applications.

Because of their interest and knowledge, citizen scientists are an excellent group to test new IEM

and monitoring technologies and processes (e.g. Smartphone apps). They can be used to vet or

advise on structured decision making and other community engagement and decision making

processes that are critical to the success of IEM applications. Finally, citizen science groups can

provide historical knowledge and local stakeholder continuity to ensure persistence and

improvement of IEM application efforts.

Removing organizational and institutional barriers to the community conduct of IEM is a

challenge. The scale at which a community of practice is achieved reflects the scale at which

IEM learning occurs and problem solutions are accessible and transferable. Currently, the scale

of community where access and transfer of solutions occurs is, at best, organizational or defined

by a particular integrated modeling domain (e.g. Table 5). Sharing high-level ideas is easier than

sharing the process that produces tangible products. Each organization or group has a mission

which defines its work, needs, and priorities. There are many fundamental differences in the

types of inter- or intra-organizational responsibilities including: regulatory/enforcement, resource

management, scientific research/monitoring, education and outreach, issue advocacy, community

40

engagement, commercial or private sector development, etc. Even when there is a joint interest

among several organizations or groups (e.g., working together to develop or apply an IEM),

differences in priorities or alignment of perspectives can create barriers to effective

collaborations. Organizations can be the greatest facilitators, or barriers, to a community of

practice approach. For the IEM community to be successful organizations must come to terms

with the need to share and collaborate in their day to day operations. This means recognizing

mutual interests and facilitating cross-organizational working relationships to pursue them.

As collaboration tools are put in place and institutional barriers are removed, the full community

moves toward participation. It is the perfect time to formalize the discipline of IEM and establish

its community identity and practices. To that end, a draft charter, describing the structure and

operation of a formal Community of Practice for IEM has been developed3. The next step is to

share this document with the larger IEM community to gain feedback and with organizational

managers to gain sponsorship.

6. THE IEM VISION AND ROADMAP TO THE FUTURE

The previous sections provided a holistic view of the IEM landscape and described, from the

current community‘s perspective, what is needed to move IEM forward. With this background

we now turn to organizing the future of integrated environmental modeling activities. Our goal is

to present a vision of IEM for the next decade and provide a roadmap that can help achieve the

vision. Section 6.1 presents the vision. The roadmap is described as a combination of cultural

changes and specific goals and activities to be pursued. The cultural changes, described in

Section 6.2, reflect a shift in the way environmental problems are defined and how solutions are

developed. The goals and activities are presented in Section 6.3; they represent the desired future

state of IEM and the steps required to achieve this state. These goals and activities represent an

integration of the information presented throughout the paper and thus include items that cross

the four principal elements of the IEM landscape.

3 http://iemhub.org/resources/382/supportingdocs

41

Three notes concerning the roadmap, first we assume that disciplinary research and development

will continue within the discipline-based communities. We do not articulate specific science

directions per discipline however IEM scientists and modelers will need to work with these

communities in an effort to efficiently access and properly integrate their science. Secondly, it is

clear that monitoring data and observations will play an important role in IEM and its

applications and supported decisions. However, the present roadmap does not attempt to define

specific monitoring based activities, but instead encourages increased collaboration between the

IEM modeling community and the monitoring community. Finally, this roadmap is analogous to

a national roadway system, as opposed to a detailed street map of cities and neighborhoods. It

does not attempt to design an implementation plan, although we believe the seeds to such efforts

are given.

6.1 Vision of the Future of Integrated Environmental Modeling

The IEM vision (Box 6) reflects the complexities of modern environmental problems and the

challenges of providing the information needed to inform decisions and policies.

Box 6. The Vision of the Future of Integrated Environmental Modeling The vision is of a future where complex environmental decisions and policies are based on systems science,

cost effective, socially responsible, adaptive, and sustainable.

Applications reflect problem formulations and solution approaches that are holistic, establishing a view of the

human-environmental system that recognizes the interdependent relationships among responsible organizations,

management scenarios (decision options), co-occurring stressors, the environment, and socio-economic

structures. This integrative systems approach engages all stakeholders throughout the decision process.

IEM Science is transdisciplinary, involving the integration of social, economic, and environmental science into

modeling systems that describe and forecast the behavior of the human-environmental system in response to

natural and human induced stress. The science provides methods for characterizing and communicating model

sensitivities and uncertainty to both decision and science stakeholders. IEM is a specific curriculum and is

taught in schools and Universities.

Information Technology provides for discovery, access, and integration of science components (data, models,

and assessment strategies) developed by stakeholders world-wide. Integrated modeling systems are constructed

and executed on a variety of platforms serving research, applications, and education.

A Community of IEM stakeholders and associated organizations engages in, invests in, and contributes to, the

shared and open development of integrated environmental science and related computer-based technologies.

Additionally, the community grows to engage interested or concerned citizens, decision makers, and their

associations and gains their support for the development and application of IEM.

42

6.2 Guiding Principles for Advancing IEM

Modern environmental problems have required fundamental adjustments in the execution of the

IEM supported decision and policy development process. In particular, the manner in which

environmental problems are perceived, articulated, assessed, and resolved is evolving rapidly. In

this section we present several key concepts, attitudes, and priorities that are driving this

evolution. We suggest that these guiding principles should form the foundation for planning and

executing IEM activities. They are closely related to one another but each possesses a defining

uniqueness that warrants attention. The guiding principles include: stakeholder involvement,

community development, systems thinking, openness, investment, and reuse.

6.2.1 Stakeholder Involvement

There is a need to expand current efforts to mobilize and involve the array of stakeholders within

the IEM process. The numerous stakeholders involved in an IEM problem and solution (Section

1.1) possess specific interests, concerns, and expertise and are principally responsible for one or

more steps in the decision process. The relationship among stakeholders is conceptually similar

to the relationship among components of the social-economic-environmental system. That is,

there is an information flow among stakeholders; each stakeholder provides information for

others and consumes information contributed by others. There is a system goal to which all

stakeholders are oriented.

Ensuring appropriate stakeholder participation, assimilating the range of stakeholder

perspectives, contributions and needs, and developing a consensus understanding of the problem,

decision goals, conceptualized system, and solutions is a significant challenge. As a principle,

full and open stakeholder involvement must be viewed as an essential ingredient for successful

decision making.

6.2.2 Community Development

There are two aspects of community development that are important to the future of IEM and

supported decisions. First, there is a need to link together the stakeholder communities

responsible for individual problems and decisions. Secondly, there is a need to link disciplinary

43

scientists and modelers with IEM scientists and modelers to form a larger community where

transdisciplinary research and development is pursued.

These communities are important because of the degree of social and technical complexity

associated with modern problems. Defining the right problem and configuring an appropriate

solution requires experience, which in turn requires time and effort. Being connected enables

experience gained addressing one problem to be available in a timely and transparent manner to

others facing similar problems.

Each of these community development efforts is an effort to form a community of communities.

At the present time we bump into each other in various venues (public meetings, scientific

conferences) but by and large we are not organized to maximize the exchange of knowledge and

experience related to IEM. Given the magnitude of the problems being faced and the

consequences of poor decisions and policies it is in our collective best interest to address the

problem of community development.

6.2.3 Systems Thinking

A systems framework for articulating, discussing, and resolving complex environmental

problems is necessary, because modern environmental problems are characterized by: (1) the

extent of the natural and built environment involved; (2) the dynamic and inter-dependent nature

of stressors and their impacts; (3) the diversity of the stakeholder community; and (4) the explicit

integration of social, economic, and political considerations. Intrinsic to complex systems are the

virtually limitless combinations of significant component interactions that may occur and the

idea of emergent properties; i.e. the properties of the system that result from interactions among

the parts. Intrinsic to solutions of complex problems is the idea that an appropriate decision (i.e.,

one that is science-based, cost effective, socially responsible, adaptive, and sustainable) is driven

by social interactions among stakeholders holding a myriad of partially compatible interests.

This is only possible in a systems framework, without which the effectiveness of decisions is left

solely to chance.

44

Systems thinking is not only a responsibility for science and technology oriented stakeholders. It

is also a responsibility for other stakeholders, including decision makers, managers, assessors,

the public, industry representatives, etc. Understanding and appreciating how the social-

economic-environmental system works and how individual interests, concerns, or processes fit

into the larger system are key to achieving the IEM vision. This has several consequences. First,

movement toward systems thinking means that every IEM related activity, no matter how small,

is conceptualized, designed, and executed with full consideration of the relevant system of

interacting parts. Second, all interests, preferences, and concerns are viewed together. Third,

trade-offs and compromises are explicitly recognized and tracked as an intrinsic responsibility of

IEM-based decision making.

There are no unique barriers to this undertaking. Progressive, and integrated, systems thinking

are natural to human development. Our lack of knowledge and experience with this topic,

combined with the scale and scope of consequences of poor decisions and policies, simply makes

the process more challenging and urgent.

6.2.4 Openness

The IEM community must embrace and encourage openness, cooperation, and collaboration as it

relates to the planning and execution of IEM activities. The greater IEM community presently

shows a high degree of openness and willingness to share ideas. The process by which product

development is initiated and executed, however, remains relatively closed. Products (data,

models, methods) are typically made available by their developers only upon completion and/or

publication. Ironically, once completed, it is common for developers to invite collaboration on

further development. Groups and organizations are less inclined to accept these delayed

invitations and more inclined to conduct their own in-house development (often aimed at nearly

the same product functionality).

Why does this happen and can it be changed? An unflattering explanation is the ‗not invented

here‘ syndrome; everyone wants to be in the driver‘s seat. Given the choice to lead (and control)

or collaborate (and contribute), we often choose to lead. Another explanation is that sharing of

information, especially with a large community where personal relationships are not well

45

established, often does not fully recognize (and reward) the efforts and the leadership of those

who have contributed. Such sharing may also result in a loss of intellectual property rights or in a

loss of economic development potential by a given party. Yet another possible explanation is that

conducting development that is dependent on timely and quality execution by external parties

often incurs risks and uncertainty that would not exist if the development is conducted ―in-

house‖. Uncertainties and risk exist because existing products are not sufficiently accessible and

documented to allow timely adoption for a new, albeit similar, purpose. The reality is most likely

that each of these considerations contribute to an explanation why openness is lacking.

Despite these issues, progress can be made. There are an increasing number of open source

software development projects that attract voluntary contributions from a world-wide

community. From these experiences we know that under some conditions, open, cooperative,

collaborative efforts do work. The future of IEM depends on identifying processes and

mechanisms that overcome the barriers and encourage openness and collaboration.

6.2.5 Reusable products

Across the world there is a large community of scientists, modelers, and technologists

developing databases, modeling components, and infrastructure for IEM. Our collective ability to

efficiently share and reuse the products of these efforts remains limited. Reuse can be achieved at

different levels. On the high end, reusable means an IEM component can be accessed and reused

in a context other than its original development without additional effort or reusable ―as is‖.

From a more practical perspective, reusability means that only a fraction of the original effort is

required in order to assimilate and utilize the component or application, and that the re-user is

certain of knowing the science basis of the application or component, its complete functionality,

and setup (i.e., assumptions and parameterizations).

There are real benefits to rethinking the way we build, execute, and publish the applications,

science, and technology of IEM to optimize reuse and adaptation:

More cost effective assessments.

More decisions informed with science-based integrated modeling.

46

Increased consistency in the application of science. This, in turn, increases the

effectiveness of communication among stakeholders.

A natural evolution, identification, promotion, and sharing of best practices related to

applications, science, and technology.

Increased ease of review and implementation of quality controls and audits for IEM and

its applications; and consequently increased credibility for IEM-supported decision

making.

Ability to extend the life of a solution and employ adaptive management strategies (e.g.,

by collecting new data and updating the solution and management strategy.

Ability to assimilate solution components from multiple developing organizations to form

solutions to new problems

Increased research opportunities. Using software tools researchers could access archived

applications/solutions and assess and improve the modeling process itself.

6.2.6 Investment

Infrastructure to facilitate IEM is currently limited to scales that serve groups or organizations

that have resources to develop IEM systems. These systems are generally focused on specific

applications of importance to the organization. The science and technology contained in these

systems, however, is applicable to many other problems in other organizational contexts. It

remains relatively inaccessible due to a combination of incompatible technologies and a lack of

documentation. For IEM to effectively serve decisions and policies in the 21st century, strategic

investments are required in infrastructure to facilitate the wider community‘s ability to discover,

retrieve, integrate, and apply the products of IEM science. Similarly, strategic investments

should focus on the development of human-based and technology facilitated infrastructure for

engaging IEM stakeholders in the greater IEM process, including problem definition, scenario

development, data gathering, modeling, reviews, and decision making frameworks. In essence,

IEM and related decision processes need to become a recognized and supported science

discipline. Until these investments become a specific community priority, IEM will remain an ad

hoc solution to an expanding inventory of complex problems.

6.3 The IEM Roadmap

47

This section reviews the needs expressed throughout the paper and organizes them into a

coordinated, coherent set of goals and related activities. The goals define desired endpoints and

the activities identify potential steps to take in order to achieve them. Some activities standalone,

that is, they are not dependent on information or products from other tasks. Most activities,

however, have dependencies and require a certain sequencing.

In figures 5 through 9, activity sequencing progresses from the bottom to the top of the figure.

Each activity is labeled with icons representing which landscape elements are involved (i.e.,

application, science, technology, community). The first icon listed represents the primary

element involved and the others contribute important information, expertise, and perspective.

Thus, for example, to achieve the goals relevant to IEM applications may require the integration

of efforts across science, technology, and community elements. The roadmap goals and activities

are summarized below.

Goal 1: Establish and promote both the science discipline of IEM and a formal community

of practice. The activities required to achieve this goal focus on: defining the science and

community domains; establishing appropriate roles and relationships among the many

stakeholders; establishing a web based technology to facilitate interactions involving a globally

dispersed community; and conducting several activities oriented toward outreach and

establishing community culture and practices. The sequencing of activities is presented in Figure

5.

Goal 2: Provide community access to tools for designing, executing, documenting, and

archiving IEM applications and modeling science. The desire is to move beyond the relatively

limited accessibility and usability of IEM science and technology to dramatically increasing

collaboration among developers and ease of use for others. There are two related activity tracks

(Figure 6): one focused on the science and technology of decision support systems and another

focused on quantitative modeling. The activities required to achieve these goals are similar and

include developing a series of standards designed to facilitate the development and publication of

reusable IEM technologies.

48

Goal 3: Advance the state-of-the-art of IEM applications. Activities supporting this goal

(Figure 7) include developing better methods for involving all stakeholders in the application

process, advancing the decision making process by further developing adaptive management

strategies and methods, developing methods for peer reviewing complex applications, and using

IEM applications and vehicles for educating stakeholders.

Goal 4: Advance the science of IEM. A principle focus revolves around the need to apply

systems thinking to the process of building modeling solutions for complex problems. Important

considerations are being explicit and rigorous in developing problem statements and

management scenarios and building appropriately complex conceptualizations of the system.

Other important science activities (Figure 8) include: advancing practitioners access to the wide

array of data required for IEM; developing the science of information exchange among modeling

components with varying dimensional attributes; encouraging the development of thorough peer

review protocols; and developing standards methods for addressing model evaluation and

uncertainty.

Goal 5: Establish software development protocols and explore innovative technologies to

enhance the development and application of IEM. The supporting activities (Figure 9) are

specific to advancing the technological means for developing, documenting, testing, reviewing,

and archiving products ranging from small local databases to science algorithms of an

environmental process to full modeling systems to full applications. It will be necessary to

develop standardized means for versioning software, discovering and integrating components,

publishing software for reuse, and exploring innovative means of leveraging the internet and

other emerging social media for the purpose of extending the overall effectiveness of IEM in the

decision making context.

7. CONCLUDING STATEMENT

This paper has described the current state and understanding of IEM. Based upon a knowledge of

what has already been accomplished and on the results of a number of workshops that were

critical to the IEM community, it has proposed a roadmap for the future. The main elements of

49

the roadmap can be split into applications, science, technology and community. The hope is that

within the next decade there is a vibrant, connected international community that will have the

tools and techniques by which to routinely undertake IEM and effectively interact with

stakeholders. When the goals for our roadmap are achieved, IEM will be accessible to a wider

community of users and a new generation of research and development will be enabled. We also

envison that environmental management and regulation decisions at governmental levels can

become increasingly informed by IEM applications. With greater sharing, hard work, and

appropriately directed resources, this vision is achievable.

50

Figure 5: IEM Goal 1

51

Figure 6: IEM Goal 2

52

Figure 7: IEM Goal 3

53

Figure 8: IEM Goal 4

54

Figure 9: IEM Goal 5

55

Disclaimer

The views expressed in this paper are those of the authors and do not necessarily reflect the

views or policies of the affiliated organizations.

References

ABARE-BRS (Australian Bureau of Agricultural and Resource Economics – Bureau of Rural

Sciences), 2010. Assessing the regional impact of the Murray–Darling Basin Plan and the

Australian Government‘s Water for the Future Program in the Murray–Darling Basin.

ABARE–BRS Project 43051. Commonwealth of Australia. Canberra, ACT, Australia.

Akbar, M., Aliabadi, S., Patel, R., Marvin, W., 2011. Fully Automated and Integrated Multi-

Scale Forecasting Scheme for Emergency Preparedness. Environ. Model. Software (this

issue).

Ahuja, L.R., Ascough II, J.C., David, O., 2005. Developing natural resource models using the

object modeling system: feasibility and challenges. Adv. Geosci. 4, 29–36.

Arciniegas, G., Janssen, R., Rietveld, P., 2011. Effectiveness of collarborative map-based

decision support tools: results of an experiment. Environ. Model. Software (this issue).

Ascough II, J.C., Maier, H.R., Ravalico, J.K., Strudley, M.W., 2008. Future research challenges

for incorporation of uncertinaty in environmental and ecological decision-making.

Ecological Modelling 219, 383-399.

Aster, R.C., Borchers, B., Thurber, C. H., 2005. Parameter Estimation and Inverse Problems.

Amsterdam, Elsevier Academic Press.

Babendreier, J. E., Castleton, K. J., 2005. Investigating uncertainty and sensitivity in integrated,

multimedia environmental models: tools for FRAMES-3MRA. Environ. Model. Software

20 (8), 1043-1055.

Bailey, G.W., Mulkey, L.A., Swank, R.R., 1985. Environmental Implications of Conservation

Tillage: A System Approach, in: D‘Itri, F.M. (Ed.), A System Approach to Conservation

Tillage. Lewis Publishers, Inc., Chelsea, MI pp. 239-265.

Beven, K., 2007. Towards integrated environmental models of everywhere: uncertainty, data and

modelling as a learning process. Hydrol. Earth Syst. Sci. 11 (1), 460-467.

Beven, K., 2009. Environmental Modelling: An Uncertain Future? Taylor & Francis, UK.

56

Blind, M.W., Moore, R.V., Scholten, H.M., Refsgaard, J.C., Borowski, I., Giupponi, C., Estrela,

M. and Vanrolleghem, P.A., 2005a. Current results of the EC-sponsored catchment

modelling (CatchMod) cluster: Part 1: ―Cross-cutting issues‖, IWA 2005 Watershed &

River Basin Management Specialist Group Conference, Calgary, Alberta, Canada.

Blind, M.W., Borgvang, S.A., George, D.G., Froebrich, J., Zsuffa, I., Vanrolleghem, P.,

Jørgensen, L.F. & de Lange, W.J., 2005b. Current results of the EC-sponsored catchment

modelling (CatchMod) cluster: Part 2: ―Water topics and synthesis‖ IWA 2005

Watershed & River Basin Management Specialist Group Conference, Calgary, Alberta,

Canada.

Bolte, J.P., Hulse, D.W., Gregory, S.V., Smith, C., 2006. Modeling biocomplexity - actors,

landscapes and alternative futures. Environ. Model. Software 22, 570-579.

Boumans, R, Costanza, R., Farley, J., Wilson, M.A., Portela, R., Rotmans, J., Villa, F., Grasso,

M., 2002. Modeling the dynamics of the integrated earth system and the value of global

ecosystem services using the GUMBO model. Special Issue: The Dynamics and Value of

Ecosystem Services: Integrating Economic and Ecological Perspectives. Ecol. Econ. 41,

529–560.

Castronova, T.A., Goodall, J.L., 2010. A generic approach for developing process-level

hydrologic modeling components. Environ. Model. Software 25, 819–825.

Clark, M.P.,Kavetski, D., Fenicia, F., 2011. Pursuing the method of multiple working hypotheses

for hydrological modeling. Water Resour. Res. 47, W09301.

Cohen, Y., 1986. Pollutants in a Multimedia Environmental. Plenum Press, New York.

Costanza, R., A. Voinov, R. Boumans, T. Maxwell, F. Villa, H. Voinov and Wainger, L. 2002.

Integrated ecological economic modeling of the Patuxent river watershed, Maryland.

Ecological Monographs, 72 (2), pp.203-231.

Daniels, M., 1999. Integrating Simulation Technologies with Swarm, Agent Simulation:

Applications, Models and Tools Conference. University of Chicago.

David, O., Markstrom, S.L., Rojas, K.W., Ahuja, L.R., Schneider, W., 2002. The object

modeling system, in: Ahuja, L.R., Ma, L., Howell, T.A. (Eds.), Agricultural System

Models in Field Research and Technology Transfer. Lewis Publishers, Boca Raton, FL,

USA, pp. 317-344.

Delsman, J., Kukuric, N., Veldhuizen, A., Tiktak, A., Kroon, T., 2009. Netherlands Hydrological

Modeling Instrument - Coupling groundwater and surface water for national policy

analysis. In: IAHS-IAH Conference Hyderabad, 7 September 2009

Ellarby, M.J., Kite, S.S., 2006. Are Rich Visual Environments a Gimmick or a Real Aid in the

Understanding and Acceptance of Simulation Models and Results, in: Robinson, S.,

57

Taylor, S., Brailsford, S., Garnett, J. (Eds.), Proceedings of the 2006 Operations Research

Society Simulation Workshop. Leamington Spa, UK, March 28-29, 2006, pp. 295-299.

EPA (US Environmental Protection Agency). 2000. Toward Integrated Environmental Decision-

making. EPA-SAB-EC-00-011. Washington, DC. Science Advisory Board.

http://www.epa.gov/sab/pdf/ecirp011.pdf

EPA (US Environmental Protection Agency). 2004. The Integrated Environmental Strategies

Handbook. EPA430-B-04-006. Washington, DC. Office of Air and Radiation.

http://www.epa.gov/ies/documents/ies_handbook.htm

EPA (US Environmental Protection Agency), 2007. Workshop Report: Integrated Modeling for

Integrated Environmental Decision Making Workshop. January 30–February 1, 2007.

Research Triangle Park, NC http://www.epa.gov/crem/crem_integmodelwkshp.html

EPA (US Environmental Protection Agency), 2008a. Workshop Report: Collaborative

Approaches to Integrated Modeling: Better Integration for Better Decision-Making.

December 10–12, 2008. Phoenix, AZ http://www.epa.gov/crem/integrated-modeling-

workshop2008.html

EPA (US Environmental Protection Agency), 2008b. Integrated Modeling for Integrated

Environmental Decision Making. EPA-100-R-08-010. Washington, DC. Office of the

Science Advisor.

http://www.epa.gov/CREM/library/IM4IEDM_White_Paper_Final_(EPA100R08010).pd

f

EPA (US Environmental Protection Agency). 2009. Guidance on the Development, Evaluation,

and Application of Environmental Models. EPA/100/K-09/003. Washington, DC. Office

of the Science Advisor. http://www.epa.gov/CREM/library/cred_guidance_0309.pdf

European Commission, 2000, Directive of the European Parliament and of the Council

2000/60/EC Establishing a Framework for Community Action in the Field of Water

Policy, Official Journal 2000 L 327/1, European Commission, 73pp., Brussels, Belgium.

Farber, S., Costanza, R., Childers, D.L., Erickson, J., Gross, K., Grove, M., Hopkinson, C.S.,

Kahn, J., Pincetl, S., Troy, A., Warren, P., Wilson, M., 2006. Linking Ecology and

Economics for Ecosystem Management. BioScience 56 (2), 117-129.

Fenner, K., Scheringer, M., MacLeod, M., Matthies, M., McKoone, T.E., Stroebe, M., Beyer, A.,

Bonnell, M., Le Gall, A.-C., Klasmeier, J., Mackay, D., van de Meent, D.W., Pennington,

D., Scharenberg, B., Suzuki, N., Wania, F., 2005. Comparing Estimates of Persistence

and Long-Range Transport Potential among Multimedia Models. Environ. Sci. Technol.

39 (7), 1932-1942.

58

Fischenich, J.C. 2008. The Application of Conceptual Models to Ecosystem Restoration. U.S.

Army Corps of Engineers ERDC/EBA TN-08-1. ERDC Environmental Laboratory,

Vicksburg, MS.

Gijsbers, P., Gregersen, J., Westen, S., Dirksen, F., Gavardinas, C., Blind, M., 2005. OpenMI

Document Series: Part B Guidelines for the OpenMI (version 1.0). Centre for Ecology

and Hydrology, Wallingford, UK. (http://www.openmi.org/reloaded/about/documents-

publications/B_Guidelines.pdf, last accessed September 26, 2011).

Guzy, M. R., Smith, C.L., Bolte, J.P., Hulse, D.W., Gregory, S.V., 2008. Policy research using

agent-based modeling to assess future impacts of urban expansion into farmlands and

forests. Ecol. Soc. 13 (1), 37.

Hanna, S.R., 1988. Air quality model evaluation and uncertainty. J. Air Pollut. Control Assoc. 38

(4), 406-412.

Hart, B.T., Shenton, W., Chan, T., 2009. Beyesian Network Models for Einvornmental Flow

Decision-Making. Interim Final Report. Land & Water Australia Project UMO49.

Available online at http://www.sci.monash.edu.au/wsc/publications/docs/2009-interim-

final-report.pdf. Accessed December 5, 2011)

Heubusch, K. 2006. Interoperability: What it Means, Why it Matters. J. AHIMA 77 (1), 26-30.

Hill, C., DeLuca, C., Balaji, Suarez, M., da Silva, A., 2004. The architecture of the earth system

modeling framework. Comput. Sci. Eng. 6 (1), 18-28.

Hill, M.C., Tiedeman, C.R., 2007. Effective calibration of groundwater models, with analysis of

data, sensitivities, predictions, and uncertainty. John Wiley and Sons, New York.

Hughes, Andrew; Jackson, Chris; Mansour, Majdi; Bricker, Stephanie; Barkwith, Andrew;

Williams, Ann; Abesser, Corinna; Wang, Lei. 2011 Integrated modelling within the

Thames Basin : examples of BGS work. [Poster] In: Cities, catchments and coasts:

applied geoscience for decision-making in London and the Thames Basin, London, UK,

13 May 2011. (http://nora.nerc.ac.uk/14267/)

Hulse, D., Branscomb, A., Enright, C., Bolte, J., 2008. Anticipating floodplain trajectories: a

comparison of two alternative futures approaches. Landsc. Ecol. 24 (8), 1067-1090.

ICSU (International Council for Science), 2010. Regional Environmental Change: Human

Action and Adaptation. International Council for Science, Paris.

IEEE (Institute of Electrical and Electronics Engineers), 1989. IEEE Standard 610.3-1989 –

IEEE Standard Glossary of Modeling and Simulation Terminology, Institute of Electrical

and Electronics Engineers Standards Association, New York, New York.

IPCC (Intergovernmental Panel on Climate Change), 2007. Climate Change 2007: The Physical

Science Basis. Contribution of Working Group I to the Fourth Assessment Report of the

59

Intergovernmental Panel on Climate Change, in: Solomon, S., Qin, D., Manning, M.,

Chen, Z., Marquis, M., Averyt, K.B., Tignor, M., Miller, H.L. (Eds.), Cambridge

University Press, Cambridge, United Kingdom and New York, NY.

Jakeman, A. J., Letcher, R. A., 2003. Integrated assessment and modelling: features, principles

and examples for catchment management. Environ. Model. Software 18 (6), 491-501.

Jakeman, A. J., Letcher, R. A., Norton, J. P., 2006. Ten iterative steps in development and

evaluation of environmental models. Environ. Model. Software 21 (5), 602-614.

Jassby, A. D., Cloern, J. E., 2000. Organic Matter Sources and Rehabilitation of the Sacramento-

San Joaquin Delta (California, USA). Aquat. Conserv: Mar. Freshw. Ecosyst. 10, 323-

352.

Johnston, J. M., McGarvey, D. J., Barber, M. C., Laniak, G., Babendreier, J. E., Parmar, R.,

Wolfe, K., Kraemer, S. R., Cyterski, M., Knightes, C., Rashleigh, B., Suarez, L.,

Ambrose, R., 2011. An integrated modeling framework for performing environmental

assessments: Application to ecosystem services in the Albemarle-Pamlico basins (NC and

VA, USA). Ecol. Model. 222 (14), 2471-2484.

Lancaster, J.J., 2007. Integrated Monitoring, Modeling, and Management (IM3) Methodology

Applied to Governance of Global Energy Resources and Global Energy Use. Proceedings

of the Amsterdam Conference on the Human Dimensions of Global Environmental

Change ‗Earth System Governance: Theories and Strategies for Sustainability.‘

Amsterdam, The Netherlands. May 2007.

Lave, J., Wenger, E., 1991. Situated Learning: Legitimate Peripheral Participation. Cambridge

University Press, Cambridge.

Lehman, P. W., Giulianotti, J., Sevier, J., 2001. The Contribution of Algal Biomass to Oxygen

Demand in the San Joaquin River Deep Water Channel, Fall 2000 (Draft Report),‖

Department of Water Resources, Environmental Services Offices, Sacramento, CA.

Linker, L.C., Shenk, G.W., Dennis, R.L., Sweeney, J.L., 1999. Cross-Media Models for the

Chesapeake Bay Watershed and Airshed. Chesapeake Bay Program Office, Annapolis,

MD.

Litton, G. M., 2001. Sediment Oxygen Demand, Sediment Deposition Rates and Biochemical

Oxygen Demand Kinetics in the San Joaquin River near Stockton, California, Fall 1999

(Final). Report prepared for City of Stockton and San Joaquin River Dissolved Oxygen

TMDL Technical Committee, June 25 (2001).

Liu, Y., Gupta, H., Springer, E., Wagener, T., 2008. Linking science with environmental

decision making: Experiences from an integrated modeling approach to supporting

sustainable water resources management. Environ. Model. Software 23 (7), 846-858.

60

Lloyd, W., David, O., Ascough II, J.C., Rojas, K.W., Carlson, J.R., Leavesley, G.H., Krause, P.,

Green, T.R., Ahuja, L.R., 2011. Environmental modeling framework invasiveness:

Analysis and implications. Environ. Model. Software. 26, 1240-1250.

Ludascher, B., Lin, K., Brodaric, B., Baru, C., 2003. GEON: Toward a Cyberinfrastructure for

the Geosciences—A Prototype for Geologic Map Integration via Domain Ontologies.

U.S. Geological Survey Open-File Report 03-071.

Luna-Reyes, L.F., 2003. Model Conceptualization: A Critical Review, in: Eberlein, R., Diker,

V., Langer, R., Rowe, J. (Eds.), Proceedings of the 21st International Conference of the

System Dynamics Society. July 20 - 24, 2003 New York City, NY

http://www.systemdynamics.org/conferences/2003/proceed/PAPERS/236.pdf (last

accessed December 6, 2011)

Maidment, D.R., Hooper, R.P., Tarboton, D.G., Zaslaksky, I., 2009. Accessing and sharing data

using CUAHSI Water Data Services. Hydroinformatics in Hydrology, Hydrogeology and

Water Resources (Proc. of Symposium JS.4 at the Joint IAHS & IAH Convention,

Hyderabad, India, September 2009). IAHS Publ. 331, 213-223.

Maxwell, T., Costanza, R., 1995. Distributed modular spatial ecosystem modelling. Int. J. of

Comp. Simul. 5 (3), 247-262.

Mackay, D. 1991. Multimedia Environmental Models: The Fugacity Approach. Lewis

Publishers, Michigan.

Mackay, D., 2001. Multimedia environmental models: The Fugacity Approach. 2nd ed. CRC

Press, Boca Raton, Florida, USA.

Makropoulos, C., E. Safiolea, S. Baki, E. Douka, A. Stamou, and M. Mimikou, 2010. An

integrated, multi-modelling approach for the assessment of water quality: lessons from

the Pinios River case in Greece. International Environmental Modelling and Software

Society (iEMSs). 2010 International Congress on Environmental Modelling and Software

Modelling for Environment's Sake, Fifth Biennial Meeting, Ottawa, Canada.

McIntosh, B.S., Ascough, J.C., Twery, M., Chew, J., Elmahdi, A., Haase, D., Harou, J.J.,

Hepting, D., Cuddy, S., Jakeman, V., Chen, S., Kassahun, A., Lautenbach, S., Matthews,

K., Merritt, W., Quinn, N.W.T., Rodriquez-Roda, I., Sieber, S., Stavenga, M., Sulis, A.,

Ticehurst, J., Volk, M., Wrobel, M., van Delden, H., El-Sawah, S., Rizzoli, A.E. and

Voinov, A.A. (2011) Environmental Decision Support Systems EDSS development:

challenges and best practices. In: Environmental Modelling and Software, 26, 1389-1402.

McNie, E. C., 2006. Reconciling the supply of scientific information with user demands: an

analysis of the problem and review of the literature. Environ. Sci. Pol. 10, 17 - 38.

MEA (Millenium Ecosystem Assessment), 2005. Ecosystems and Human Well-being: Synthesis.

Island Press, Washington, D.C.

61

Meadows, D.H., Meadows, D. L., Randers, J., Behrens, W., 1972. The Limits to Growth.

Universe Books, New York.

Menke, W. 1989. Geophysical data analysis, discrete inverse theory, Volume 45. International

Geophyscias Series. Academic Press, Inc., San Diego, CA.

Moore, R.V., Tindall, C., 2005. An overview of the open modelling interface and environment

(the OpenMI). Environ. Sci. Pol. 8 (3), 279-286.

Moore R., Gijsbers, P., Fortune, D., Gergersen, J., Blind, M., 2005. OpenMI Document Series:

Part A Scope for the OpenMI (Version 1.0), HarmonIT.

Muetzelfeldt, R., Masheder, J., 2003. The Simile visual modeling environment. Eur. J. Agron.

18, 345 – 358.

Noth, M., Borning, A., Waddell, P., 2000. An Extensible, Modular Architecture for Simulating

Urban Development, Transportation and Environmental Impacts. Technical Report 2000-

12-01. University of Washington, Department of Computer Science.

NRC (National Research Council), 2005. Decision Making for the Environment: Social and

Behavioral Science Research Priorities. National Academies Press, Washington, DC.

NRC (US Nuclear Regulatory Commission), 2002. Proceedings of the Environmental Software

Systems Compatibility and Linkage Workshop. NUREG/CP-0177, PNNL-13654. Hosted

by The U.S Nuclear Regulatory Commission, Professional Development Center.

Rockville, MD March 7–9, 2000. http://www.nrc.gov/reading-rm/doc-

collections/nuregs/conference/cp0177/ (last accessed: December 6, 2011).

OpenMI, 2009. The Open-MI life project website http://www.openmi-life.org/ (last accessed:

December 6, 2011).

Oreskes, N., 2003. The role of quantitative models in science, in: Canham, C.D., Cole, J.J.,

Lauenroth, W.K., (Eds.) Models in Ecosystem Science. Princeton University Press,

Princeton. pp. 13-31.

Otto-Banaszak, I., Matczak, P., Wesseler, J., Wechsung, F., 2011. Different perceptions of

adaptation to climate change: a mental model approach applied to the evidence from

expert interviews. Reg. Environ. Change 11, 217–228.

Özesmi, U., Özesmi, S.L. 2004. Ecological models based on people‘s knowledge: A Multi-step

Fuzzy Cognitive Mapping Approach, Ecol. Model. 176, 43–64.

Parker, P., Letcher, R., Jakeman, A., Beck, M.B., Harris, G., Argent, R.M., Hare, M., Pahl-

Wostl, C., Voinov, A., Janssen, M., Sullivan, P., Scoccimarro, M., Friend, A.,

Sonnenshein, M., Barker, D., Matejicek, L., Odulaja, D., Deadman, P., Lim, K.,

Larocque, G., Tarikhi, P., Fletcher, C., Put, A., Maxwell, T., Charles, A., Breeze, H.,

Nakatani, N., Mudgal, S., Naito, W., Osidele, O., Eriksson, I., Kautsky, U., Kautsky, E.,

62

Naeslund, B., Kumblad, L., Park, R., Maltagliati, S., Girardin, P., Rizzoli, A., Mauriello,

D., Hoch, R., Pelletier, D., Reilly, J., Olafsdottir, R., Bin, S., 2002. Progress in integrated

assessment and modelling. Environ. Model. Software 17 (3), 209-217.

Parson, E. A., 1995. Integrated assessment and environmental policy making: In pursuit of

usefulness. Energy Policy 23 (4-5), 463-475.

Peckham, S.D., 2010. Valuation of Model Coupling Frameworks for Use by the Community

Surface Dynamics Modeling System (CSDMS).

http://csdms.colorado.edu/mediawiki/images/Peckham_2008_IGWMC.pdf (last accessed

December 6, 2011).

Quinn, N.W.T., Jacobs, K.C., 2006. An Emergency Environmental Response System to Protect

Migrating Salmon in the Lower San Joaquin River. Environ. Model. Software 22, 416-

422.

Reckhow, K.H., 2003. Bayesian approaches to ecological analysis and modeling, in: Canham,

C.D., Cole, J.J., Lauenroth, W.K., (Eds.), Models in Ecosystem Science. Princeton Univ

Press, 168-183

Refsgaard, J.C., van der Sluijs, J.P., Hojberg, A.L., Vanrolleghem, P.A., 2007. Uncertainty in the

environemtnal modeling process – A framework and guidance. Environmental

Modelling & Software 22, 1543-1556.

Robinson, S. 2006. Conceptual Modeling for Simulation: Issues and Research requirements, in:

Perrone, L.F., Wieland, F.P., Liu, J., Lawson, B.G., Nicol, D.M., Fujimoto, R.M.(Eds.),

Proceedings of the 2006 Winter Simulation Conference, 792-800.

Rotmans, J., van Asselt, M.B.A., 2001. Uncertainty Management in Integrated Assessment

Modeling: Towards a Pluralistic Approach. Environ. Monit. Assess. 69 (2), 101-130.

Rosenberg, N. J., Edmonds, J.A., 2005. Climate Change Impacts for the Conterminous USA: An

Integrated Assessment: From Mink to the ‗Lower 48‘. Clim. Chang. 69 (1), 1-6.

Saltelli, A., Ratto. M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M.,

Tarantola, S., 2008. Global sensitivity analysis, the primer: John Wiley and Sons,

Hoboken, New Jersey.

Schellekens, J., Veldhuizen, A.A., Manders, A.M.M., Winsemius, H.C. M., van Verseveld, W.J.,

te Linde, A.H., van Ulft, L.H., van Meijgaard, E., Schaap, M., Barendregt, A., Schot,

P.P., Wassen, M.J., Ward, P. J., Veldkamp, T. I. E., Mulder, H.M., Verkaik, J.,2011.

KKK-Model platform coupling. Summary report KKF01b. Deltares, TNO, Alterra, VU

Amsterdam, Universiteit Utrecht, KNMI, The Netherlands, Final Report

KFC/038A/2011, ISBN 978-94-90070-00-7.

63

Schwarz, G.E., Hoos, A.B., Alexander, R.B., and Smith, R.A., 2006. The SPARROW surface

water-quality model: Theory, application and user documentation: U.S. Geological

Survey Techniques and Methods. Book 6, Section B, Chapter 3

http://pubs.usgs.gov/tm/2006/tm6b3/. (last accessed December 6, 2011).

Sengupta, R., Bennett, D.A., 2003. Agent-based modeling environment for spatial decision

support. Int. J. of Geogr. Inf. Sci. 17 (2), 157-180.

Sidle, R.C., 2006. Field observations and process understanding in hydrology: essential components in

scaling. Hydrological Processes 20: 1439-1445.

Sleeswijk, A.W., Heijungs, R., 2010. GLOBOX: A spatially differentiated global fate, intake,

and effect model for toxicity assessment in LCA. Sci. Total Environ. 408, 2817–2832.

Sommerfreund, J.K., Gandhi, N., Diamond, M.L., Mugnai, C., Frignani, M., Capodaglio, G.,

Gerino, M., Bellucci, L.G., Giuliani, S., 2010. Contaminant fate and transport in the

Venice Lagoon: Results from a multi-segment multimedia model. Ecotoxicol. Environ.

Saf. 73, 222–230.

Stahl, C.H., Cimorelli, A., Chow, A., 2002. A new Approach to Environmental Decision

Analysis: Multi-criteria Integrated Resource Assessment (MIRA). Bull. Sci. Tech. Soc.

22 (6), 443-459

Stahl, C., Cimorelli, A., Mazzarella, Jenkins, B., 2011. Toward sustainability: A case study

demonstrating trans-disciplinary learning through the selection and use of indicators in a

decision making process, Integr. Environ. Assess. Manag. 7, 483–498.

Tapscott, D., Williams, A.D., 2006. Wikinomics: How Mass Collaboration Changes Everything,

Portfolio, New York, NY.

Tol, R.S.J., Vellinga, P., 1998. The European Forum on Integrated Environmental Assessment.

Environ. Model. Assess. 3 (3), 181-191.

Tolk, A., Diallo, S.Y., Turnitsa, C.D., Winters, L.S., 2006. Composable M&S Web Services for

Net-centric Applications. J. Def. Model. Simul. 3 (1), 27-44

Tress, G., Tress, B., Fry, G., 2005. Clarifying Integrative Research Concepts in Landscape

Ecology. Landsc. Ecol. 20 (4), 479-493.

van Evert, F.K.; Holzworth, D.; Muetzelfeldt, R.M.; Rizzoli, A.E.; Villa, F. Convergence in

integrated modeling frameworks, in: Zerger, A., Argent, R.M. (Eds). Proceedings of the

MODSIM 2005 International Congress on Modelling and Simulation. Melbourne,

Australia, December, 2005.

van Ittersum, M. K., Ewert, F., Heckelei, T., Wery, J., Olsson, J.A., Andersen, E., Bezlepkina, I.,

Brouwer, F., Donatelli, M., Flichman, G., Olsson, L., Rizzoli, A.E., van der Wal, T.,

Wien, J.E., Wolf, J., 2008. Integrated assessment of agricultural systems – A component-

64

based framework for the European Union (SEAMLESS). Agricultural Systems 96, 150-

165.

Voinov, A., Gaddis, E., 2008. Lessons for Successful Participatory Watershed Modeling: A

Perspective from Modeling Practitioners. Ecological Modelling: 216, p. 197–207.

Voinov A. 2008. Systems Science and Modeling for Ecological Economics. Academic Press,

430 p.

Voinov, A., Bousquet, F., 2010. Modelling with stakeholders. Environ. Model. Software 25 (11),

1268-1281.

Voinov, A., and C. Cerco. 2010. Model integration and the role of data. Environmental

Modelling & Software 25, no. 8: 965-969.

Voinov, A., Shugart, H., 2011. ―Integronsters‖, integral and integrated modeling. Environmental

Modelling and Software (this issue).

Voinov, A., Hood, R.R., Daues, J.D., Assaf, H., Stewart, R., 2008. Building a community

modelling and information sharing culture, in: Jakeman, A.J., Voinov, A., Rizzoli, A.E.,

Chen, S.H. (Eds.) State-of-the-Art and Futures in Environmental Modelling and

Software. Elsevier, New York. pp. 345–365.

Voinov, A., D. Arctur, I. Zaslavskiy, and S. Ali, 2008. Community-based software tools to

support participatory modelling: a vision. In: M. Sànchez-Marrè, J. Béjar, J. Comas, A.

E. Rizzoli, G. Guariso (Eds.) Proceedings of the iEMSs Fourth Biennial Meeting:

International Congress on Environmental Modelling and Software (iEMSs 2008).

Voinov, A., DeLuca, C., Hood, R.R., Peckham, S., Sherwood, C.R., Syvitski, J.P.M., 2010. A

Community Approach to Earth Systems Modeling. EOS, Trans. Am. Geophys. Union 91

(13), 117-124.

Wang, W.G., Tolk, A., Wang, W.P. 2009. The Levels of Conceptual Interoperability Model:

Applying Systems Engineering Principles to Modeling and Simulation, in: Proceedings

of the Spring Simulation Multiconference. Society for Modeling & Simulation

International (SCS), San Diego, CA.

Weyant, J. P., Davidson, O., Dowlatabadi, H., Edmonds, J.A., Grubb, M., Parson, E.A., Richels,

R., Rotmans, J., Shukla, P.R., Tol, R.S.J., Cline, W.R., Fankhauser, S., 1996. Integrated

assessment of climate change: An overview and comparison of approaches and results,

in: Bruce, J.P., Lee, H., Haites, E.F., (Eds.), Climate Change 1995: Economic and Social

Dimensions of Climate Change Contribution of Working Group III to the Second

Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge

University Press, New York City, NY, pp. 368-396.

65

Zagonel, A.A., 2002. Model Conceptualization in Group Model Building: A Review of the

Literature Exploring the Tension Between Representing Reality and Negotiating a Social

Order. Proceedings of the 2002 International Conference of the System Dynamics

Society. Palermo, Italy (July 28 – August 1).