1

Ecomodel Team Responses to HCP Science Committee Input March 3, 2015

On behalf of the HCP Ecomodel Team, I would like to thank the HCP Science Committee for your timely and well constructed comments following our February 11th

presentation and subsequent EAA request. We have summarized our decisions to the questions posed in the EAA request and have attempted to address several of the additional comments by means of the responses below. Additionally, we have a manuscript nearing completion for the multinomial logit approach (should be available for review in the April/May timeframe) and will distribute to the committee upon completion and internal review. We look forward to continued interaction with the HCP Science Committee and welcome additional comments, questions, and guidance as you see most appropriate.

1.) Does the model need to be expanded spatially? In summary, a range of responses were provided by the Science Committee from “don’t expand until proof of concept for the network of models has been provided” to “model the majority of the San Marcos River for Texas wild-rice”. Overall, the majority opinion was that some level of expansion should be conducted but the entirety of either system was likely not necessary. Upon consideration and contemplation of these comments, the project team made the decision to propose the following for the Year 3 scope submitted to EAA: Comal System: Expand to include the Upper Spring Run study reach and Landa Lake study reach. The Upper Spring Run reach is the most likely reach to first experience impacts related to low-flow conditions and has already experienced flow-related impacts during recent drought conditions. The Landa Lake reach has been the most stable habitat over the last 15 years and is presumed to remain the same as it will likely be the last water body protected under extremely low-flow conditions. As the Ecomodel objective is to test the applicability and protection of the HCP flow regime, both ends of the spectrum appear most appropriate. San Marcos System: Expand to include the I35 study reach. This allows for one study reach upstream of Rio Vista Dam (City Park reach- in progress) and one below. The upstream reach provides an index for conditions experienced more near Spring Lake dam including more consistent water quality yet a high level of recreational activity. The downstream reach (I35) provides conditions further away from the source including increased water temperatures, increased turbidity, etc. with somewhat lower recreational pressure.

2.) Period of simulation for the water quality model – how much data (i.e., number of years) is needed for calibration?

The period for water quality modeling was chosen based on available empirical data to allow sufficient time/conditions to split the record into a calibration and then validation time period with a suitable range of flow conditions within each split.

ATTACHMENT 8

2

ADDITIONAL RESPONSES: Charlie Krietler: An “Agent-base Modeling” approach is being developed to simulate future survivability of the various plant and animal species. An admirable task, but I am concerned if this modeling approach is standard to the biologic scientific community. Team Response: Agent-based/individual-based modeling (henceforth IBM) is a simulation-based approach that ecologists have used to explore the properties of ecosystems for over 40 years (Newnham 1964; Kaiser 1974, as cited in Grimm and Railsback, 2007). Grimm (1999) reviewed over 50 IBMs that have been used in ecology, and significantly more have been published in the last 16 years. During the initial stages of this project, we reviewed several different types of modeling approaches (ranging from analytical and statistical techniques to system-dynamics-based simulations), and decided that a spatially-explicit IBM was the best option for the current project for the following reasons:

1) We are exploring how fountain darter dynamics respond to spatial and temporal changes in their habitat, including vegetation growth and dispersal. Spatially-explicit IBMs explicitly provide tools to depict spatial and temporal relationships between individual organisms and their environment.

2) There is significant variability among individuals in a population, and in order to develop an understanding of how this variability affects population dynamics, we needed an approach that allows us to exhibit that variability. IBMs are programmed in an object-oriented language that allows individuals to be parameterized differently (e.g., one darter can be 3 days old and male, and another can be 1 year old and female; or each female can have a different fecundity value, etc.).

3) IBMs can be designed to embrace system complexity rather than suppress it (or, worse, ignore it).

4) IBMs, like the one being developed here, provide a tool to inform a spectrum of management possibilities as they allow researchers/natural resource managers to devise scenarios that they may not be able to examine through field experiments or changes in flow scenarios. Using an IBM, experts can develop hypotheses as to how an ecological process affects a species or how an individual would behave under a set of conditions. These testable hypotheses can provide the foundation for successful conservation.

5) Our approach closely follows the recommendations by Rose et al (2015, attached) for modeling practices for assessing the effects of ecosystem restoration on fish.

References for IBMs:

• Grimm, V. 1999. Ten years of individual-based modelling in ecology: What have we learned and what could we learn in the future? Ecological Modelling, 75-76: 641-651

• Grimm V. & Railsback S.F. (2005) Individual-based Modeling and Ecology. Princeton University Press, Princeton.

ATTACHMENT 8

3

• Grimm V., Revilla E., Berger U., Jeltsch F., Mooij W.M., Railsback S.F., Thulke H.H., Weiner J., Wiegand T. & DeAngelis D.L. (2005) Pattern-oriented modeling of agent-based complex systems: lessons from ecology. Science 310, 987-91.

• Jeltsch F., Wissel C., Eber S. & Brandl R. (1992) Oscillating dispersal patterns of tephritid fly populations. Ecological Modelling 60, 63-75.

• Mazaris A.D., Broder B. & Matsinos Y.G. (2006) An individual based model of a sea turtle population to analyze effects of age dependent mortality. Ecological Modelling 198, 174-82.

• Railsback S.F. & Grimm V. (2012) Agent-based and individual-based modeling: a practical introduction. Princeton University Press, Princeton, NH.

• Railsback S.F. & Harvey B.C. (2002) Analysis of habitat-selection rules using an individual-based model. Ecology 83, 1817-30.

• Rossmanith E., Blaum N., Grimm V. & Jeltsch F. (2007) Pattern-oriented modeling for estimating unknown pre-breeding survival rates: The case of the Lesser Spotted Woodpecker. Biological Conservation 135, 555-64.

• Swannack T.M., Grant W.E. & Forstner M.R.J. (2009) Projecting population trends of endangered amphibian species in the face of uncertainty: A pattern-oriented approach. Ecological Modelling 220, 148-59.

• Uchmanski J. & Grimm V. (1996) Individual-based modelling in ecology: what makes the difference? Trends in Ecology & Evolution 11, 437-41.

• Wiegand T., Jeltsch F., Hanski I. & Grimm V. (2003) Using pattern-oriented modeling for revealing hidden information: a key for reconciling ecological theory and application. Oikos 100, 209-22.

ATTACHMENT 8

4

Conrad Lamon: Comments/questions/requests on the FD multinomial logit as presented

Why were the count data aggregated? If it was really to deal with zeros, there are better ways. We want to be able to predict abundance, do we not? I have seen good applications of a multinomial model used with relative abundances between taxa, but for abundance (count data) I think there are better options. I see 340 observations for SM system and 795 for Comal. When adding all the observations in the categories on the page before I get 1672. Where are the other 537 observations? Would it be possible (easy?) to see a histogram of the data, with individual bins for each integer from 0 to the max. count? By system? Together? These maybe already available, as data plots are among the first steps of a statistical data analysis. It seems that the macro and micro environmental variables approach was an attempt to model Hierarchical structure. See Gelman and Hill 2007and/or, Lamon and Qian 2008, for a multilevel approach. Slide 59 presents only the AIC values of the best (smallest AIC) model, i.e. the result of the model search. It is much more informative to see the AIC of all the models attempted in the model search (even if, or perhaps especially if, some of them did no converge).In fact, a single, isolated AIC result has no meaning without another with which to compare (that was fit to the same data). Slide 60 shows a pvalue column for each of the included coefficients of the linear model portion of the logit model. The Comal results report numbered columns (2-4), and for SM (2,3). What are they and why aren’t there the same columns for each model? One must be careful not to interpret coefficients in the same way as for a linear regression with a scalar response. The coefficients are not linearly related to abundance, but to log ratios. Team Response:

1. We are aware that multinomial models (including Poisson regression, negative binomial regression and zero-inflated count models) are among the most commonly used analytical techniques for dealing with count data, such as fisheries abundances, and there also are other regression methods applicable to fishery management problems (e.g., multiple linear regression models). An alternative to the count regression models, which has the potential to offset a portion of local variation unaccounted for by the count regression models, is multinomial logit regression. In addition, the multinomial logit regression in our study provides a location (i.e., a cell) with a probability of being in any of four categories and hence provides stochasticity in our simulation model.

2. We are sorry to create the confusion. There are a total of 1,672 survey samples. For purposes of the proof of concept analysis and to rapidly get ready for the February 11th presentation, only the 100% complete files in the database were utilized. Hence, when we developed the analyses for the February 11th presentation, we only included 340 observations for San Marcos Spring and 795 for Comal Spring. Histograms of those data are shown below:

ATTACHMENT 8

5

We are diligently going back through the database for those remaining 537 samples and filling in missing values from the original data sheets (where possible) and rectifying any discrepancies, but this could not be completed in time for the presentation.

3. Yes, we considered developing a hierarchical model. However, we did not because of lack of macro-environmental information.

0

20

40

60

80

100

0 2.

5 5 7.

5 10

12.5

15

17

.5

20.5

23

25

.5

28

30.5

33

.5

36.5

40

.5

44.5

48

55

.5

61.5

70

.5

85

Cou

nts

Density (#/m2)

Comal Spring

0 5

10 15 20 25 30 35

0 1.

5 3 4.

5 6 7.

5 9 10

.5

12

13.5

15

17

19

21

23

.5

26

38

Cou

nts

Density (#/m2)

San Marcos Spring

ATTACHMENT 8

6

4. Variable selection process. Variables once removed were not returned to the model. The minimum value of AIC is in bold. Comal Spring: Model ID Variable removed AIC

a None 1608.649 b Green_Algae 1602.651 c VegCover 1596.703 d VegHeight 1591.225 e Velocity 1586.141 f DO 1583.038 g Temperature 1583.471

San Marcos Spring: Model ID Variable removed AIC

a None 872.907 b WithBryo 867.298 c Potamogeton 867.298 d Vallisneria 869.777 e Sagittaria 869.154 f MainVegHeight 866.423 g DO 863.538 h WaterDepthFt 866.035

5. Once again, we are sorry that our presentation created confusion. Somehow, in the haste

of preparation, the tables were not complete. The complete results are shown below. Comal Spring:

p-value 2 3 4

Constant - 7.7584 9.5793 2.1008 Bryophytes <.0001 4.3455 4.2244 9.8051 Cabomba <.0001 4.4947 3.3249 8.6736 Ceratopteris 0.0329 3.4015 1.8596 -5.4335 FilamentousAlgae <.0001 6.0201 4.7125 12.1659 Hygrophila <.0001 3.5061 2.8677 6.66 Ludwigia <.0001 3.9065 3.8657 8.1887 Sagittaria 0.0126 2.2736 1.2025 5.7577 Vallisneria 0.0005 3.0407 1.2385 6.8683 WithBryo <.0001 1.8536 1.935 2.8385 WaterDepthFt 0.0326 -0.3647 -0.3881 -0.0018 SpCond 0.0483 -0.0018 -0.00094 0.00022 pH <.0001 -1.4116 -1.7139 -1.6568

ATTACHMENT 8

7

San Marcos Spring:

p-value 2 3 4

Constant - -10.171 -26.6053 -26.7749 Cabomba 0.0048 3.3529 3.4414 1.8017 Hydrilla 0.0881 2.2244 1.8748 0.5355 Hygrophila 0.0069 2.9458 1.6812 -0.1177 POT_HYG 0.0094 2.7433 2.0642 -1.1731 MainVegPer 0.0827 -0.00581 0.0559 0.0717 WaterDepthFt 0.1403 -0.1848 0.2231 0.1977 Velocity 0.041 -3.0728 -4.7604 -9.6547 Temp 0.0244 0.3914 0.5201 0.4473 SpCond 0.0219 -0.00381 -0.00317 -0.0037 pH 0.0586 0.3487 1.1541 1.3725

We are aware that the coefficients should not be interpreted in the same way as for a linear regression, and will make this clear in future presentations. For example, a 1-unit increase in Bryophtes signifies that probability of category 4 is exp(9.8051) times more likely than before, after controlling for the other variables.

In conclusion, we would like to remind the Committee that, while the completed coupled model for vegetation and fountain darter population offers the potential of providing a versatile tool for investigating a range of management and scientific questions concerning the fountain darter, our central objective for the present work is to bring the model development to the level of being able to address whether the HCP flow regime is adequate for protection of the fountain darter population. In order to achieve this objective within the available time frame, it is necessary to focus on those aspects of model capability that form the critical path to determining the effect of spring flows on fountain darter population. There are many aspects of the formulation of the model and the range of conditions for which it is applicable that must await future additional research and development (if necessary).

ATTACHMENT 8

Pe

KWa

b

c

d

e

f

g

a

ARR2A

KERFCMB

1

u(t

dt(

h0

Ecological Modelling 300 (2015) 12–29

Contents lists available at ScienceDirect

Ecological Modelling

j ourna l h omepa ge: www.elsev ier .com/ locate /eco lmodel

roposed best modeling practices for assessing the effects ofcosystem restoration on fish

enneth A. Rosea,∗, Shaye Sableb, Donald L. DeAngelis c, Simeon Yurekd, Joel C. Trexlere,illiam Graf f, Denise J. Reedg

Department of Oceanography and Coastal Sciences, Energy, Coast, Environment Building, Louisiana State University, Baton Rouge, LA 70803, United StatesDynamic Solutions, 450 Laurel Street, 1060 North Tower, Baton Rouge, LA 70801, United StatesSoutheast Ecological Science Center, U.S. Geological Survey, Biology Department, University of Miami, PO Box 249118, Coral Gables, FL 33124, United StatesBiology Department, University of Miami, PO Box 249118, Coral Gables, FL 33124, United StatesDepartment of Biological Sciences, Florida International University, 3000 N. E. 151st Street, North Miami, FL 33181, United StatesDepartment of Geography, University of South Carolina, Callcott, Room, 230, Columbia, SC 29208, United StatesThe Water Institute of the Gulf, One American Place, 301 N. Main St., Suite 2000, Baton Rouge, LA 70825, United States

r t i c l e i n f o

rticle history:eceived 6 November 2014eceived in revised form2 December 2014ccepted 23 December 2014

eywords:cosystemestorationishontroversyodels

est practices

a b s t r a c t

Large-scale aquatic ecosystem restoration is increasing and is often controversial because of the eco-nomic costs involved, with the focus of the controversies gravitating to the modeling of fish responses.We present a scheme for best practices in selecting, implementing, interpreting, and reporting of fishmodeling designed to assess the effects of restoration actions on fish populations and aquatic food webs.Previous best practice schemes that tended to be more general are summarized, and they form thefoundation for our scheme that is specifically tailored for fish and restoration. We then present a 31-stepscheme, with supporting text and narrative for each step, which goes from understanding how the resultswill be used through post-auditing to ensure the approach is used effectively in subsequent applications.We also describe 13 concepts that need to be considered in parallel to these best practice steps. Examplesof these concepts include: life cycles and strategies; variability and uncertainty; nonequilibrium theory;biological, temporal, and spatial scaling; explicit versus implicit representation of processes; and modelvalidation. These concepts are often not considered or not explicitly stated and casual treatment of themleads to mis-communication and mis-understandings, which in turn, often underlie the resulting contro-

ATTACHMENT 8

versies. We illustrate a subset of these steps, and their associated concepts, using the three case studiesof Glen Canyon Dam on the Colorado River, the wetlands of coastal Louisiana, and the Everglades. Useof our proposed scheme will require investment of additional time and effort (and dollars) to be doneeffectively. We argue that such an investment is well worth it and will more than pay back in the longrun in effective and efficient restoration actions and likely avoided controversies and legal proceedings.

© 2015 Elsevier B.V. All rights reserved.

. Introduction

Large-scale aquatic ecosystem restoration is increasingly being

sed to offset or compensate for impacts made to the environmentBullock et al., 2011; Suding, 2011). Prominent examples includehe removal of dams on the Klamath River to enhance salmon

∗ Corresponding author. Tel.: +1 225 578 6346; fax: +1 225 578 6513.E-mail addresses: karose@lsu.edu (K.A. Rose), ssable@dsllc.com (S. Sable),

on deangelis@usgs.gov (D.L. DeAngelis), syurek@bio.miami.edu (S. Yurek),rexlerj@fiu.edu (J.C. Trexler), graf@sc.edu (W. Graf), dreed@thewaterinstitute.orgD.J. Reed).

ttp://dx.doi.org/10.1016/j.ecolmodel.2014.12.020304-3800/© 2015 Elsevier B.V. All rights reserved.

populations (US DOI, 2012), ensuring sufficient freshwater flowsfor biota in the Everglades (NRC, 2012a) and the California Delta(NRC, 2012b), reducing nutrient loadings to improve water qualityin the Chesapeake Bay (NRC, 2011), and offsetting the losses of wet-lands in coastal Louisiana (Peyronnin et al., 2013). Because of thelarge magnitude of the restoration actions needed and their broadspatial extent, these large-scale projects are considered expensiveand often controversial. Recent restoration plans are estimated atabout 25 billion dollars over the 50 years for the California Delta

(CADWR, 2013), 13 to 15 billion dollars to Maryland alone for theChesapeake Bay (Gray, 2013), and 20 to 50 billion dollars for theLouisiana coast-wide plan (Peyronnin et al., 2013). Actions as part ofthe recovering the Delta smelt, a U.S. Federally endangered species,

l Mod

ierra

cdmpwala

vtitSafitbatoroatTpro

tohN2mcidh“afiuamts

ebaMbacmtbt

MEN

K.A. Rose et al. / Ecologica

n the California Delta have been debated in Federal court (McKinleyt al., 2011) as a result of the uncertainty in the effectiveness ofestoration actions and their high monetary costs associated withestoration resulting in limitations on water exports for human andgricultural use (NRC, 2010).

An extreme case of a modeling melt-down was the salmon lifeycle population analyses done for evaluating the removal of theams on the Klamath River (US DOI, 2012). One of the populationodels was removed from consideration just before a peer review

anel meeting (Atkins, 2011), how the conclusions of a review panelere subsequently reported by a federal agency was reviewed by

nother panel (Atkins, 2012), and staff scientists in a Federal agencyitigated against their supervisor based on their involvement with

salmon population model (Nature Newsblog, January 9, 2013).While there are many aspects to restoring the ecosystem ser-

ices of these restoration projects, the focus often gravitates tohe responses of fish and shellfish (referred to as fish here). Thiss because of the recreational and commercial value of some ofhe fish species, their listing as endangered under the Endangeredpecies Act, and because trends in fish abundances (often indices)re visible to the interested public. The controversy arises becausesh species have complex life cycles and can be difficult to moni-or, and thus predicting their responses using modeling is necessaryut also highly uncertain (Rose, 2000). Often, only certain life stagesre within the influence of the restoration actions so that long-termrends in abundances can be greatly influenced by factors outsidef the influence of the restoration. Also, many restoration projectsesult in changes to the environment that have multiple effectsn the vital rates of the fish. For example, restoring hydrology canffect the water quality, productivity of the food base, access to cer-ain habitats, and changes to physical habitat that provide shelter.hese changes to the habitat and food can, in turn, have a com-lex mixture of effects on fish via affecting their growth, mortality,eproduction, and movement. In addition, restoration actions oftenccur in a subset of the habitats inhabited by the organisms.

Given the complexity of the situation, a common approach iso use habitat suitability indices (HSI) to assess restoration effectsn fish (e.g., Fuller et al., 2008; Nyman et al., 2013). HSI modelingas many advantages but also some key weaknesses (Ahmadi-edushan et al., 2006; Draugelis-Dale, 2008; Elith and Burgman,003; Gore and Nestler, 1988; Roloff and Kernohan, 1999). Theain advantage to a habitat-based approach is that one avoids the

hallenges in modeling fish population and community dynam-cs, which is subject to debate about the model formulation, isata-intensive, and can be highly uncertain. Habitat is critical toealthy and productive fish populations, and so determining howrestoration actions” will affect habitat relative to “no action” isn important step toward quantifying the ecological benefits tosh of restoration actions. HSI models are also relatively easy tonderstand and explain. The major disadvantage to habitat-basedpproaches is simply that they quantify habitat, which may oray not be directly correlated to fish abundance and provides lit-

le information beyond changes in habitat capacity for certain lifetages.

In some situations, there is pressure from stakeholders and oth-rs to go beyond habitat suitability to predicting the abundance andiomass responses of fish species in order to justify the restorationctions (i.e., changes in habitat capacity from HSI are not sufficient).odels of fish population and community dynamics can, in theory,

e used to assess the net population responses by attempting toccount for the full life cycles and the complex suite of effects onertain life stages and in certain areas (Rose et al., 2009). How fish

ATTACH

odels that are used to assess the large-scale effects of restora-ion are selected, implemented, interpreted, and reported thereforeecomes especially critical to ensure the credibility of the restora-ion decisions that rely on the modeling results.

elling 300 (2015) 12–29 13

One challenge is that fish modeling is a scientific process thatinvolves the judgment of the modeler. While this is true of all mod-eling, it is particularly apparent with fish and ecological modeling.Other modeling disciplines also involve judgment but usually thejudgment is more focused on the details. For example, statisticalmodeling uses data to determine which model is best, and debatesgravitate to details on which data transformation to use and deter-mining outlier points. All hydrodynamics models solve the samebasic set of fundamental physics equations (i.e., conservation ofmass and continuity of momentum), and the major judgment deci-sions are how to set up the model grid and how to deal with subgridscale processes (e.g., turbulence). Fish modeling often does not havesufficient data to identify the optimal model formulation, and fishmodeling does not have fundamental equations like hydrodynam-ics. Thus, decisions about the level of detail of processes to includein fish models get pushed more toward the judgment of the modeler(i.e., “the art of modeling”). The strong role of the modeler’s judg-ment in fish modeling does not weaken the power and utility offish modeling, but does make model selection and implementationmore challenging to document and justify.

In this paper, we present a scheme of best practices for usingmodels to assess fish responses to restoration actions. While therehave been multiple “best practice” schemes proposed for ecologicalmodeling in general (e.g., Jakeman et al., 2006), our experience isthat none of them alone are sufficiently tailored for use with fishand restoration. We first summarize previously proposed model-ing best practice schemes as a basis, and then present our versionfor fish modeling applied to restoration projects. We also describea set of concepts about fish modeling that are often at the centerof misunderstanding and controversy. We propose that combin-ing our version of the modeling steps with these concepts wouldbuild consensus about the fish modeling and thereby lead to moreeffective and less controversial restoration decisions. We illustrateseveral of the key steps and concepts using our experience withcoastal Louisiana, Florida Everglades, and the Colorado River. Whileour focus is predicting the effects of restoration on fish, these stepscan be easily applied to other taxa and ecological modeling thatdeals with evaluating future scenarios. We conclude with a discus-sion of the importance of using best practices, and some additionaladvice about how to implement the best practices framework.

2. Steps in best modeling practices

2.1. Previously proposed schemes

The idea of specifying a series of steps that would promote andencourage successful and informative ecological and environmen-tal modeling has a long history. Often, the way ecological and fishmodel analyses are presented can create the appearance that themodel was selected arbitrarily or in an ad hoc manner. Furthermore,usually only the final model structure, and a subset of the resultsof the final model, are presented. The analysis is then viewed inisolation, without the benefit of knowing how and why the partic-ular model, from the many possible models, was selected and howdecisions were made about its implementation and interpretation.Despite the appearances, models used by experienced modelers arenever arbitrarily selected. There is a careful evaluation and thoughtprocess involved in selecting a model, implementing it, and report-ing the results. However, this thought process and decision makingis rarely sufficiently documented.

A variety of best practices schemes have been proposed.

T 8

Schmolke et al. (2010) discussed ecological models supportingenvironmental decision making. They summarized from the lit-erature the elements of good modeling practice: inclusion ofstakeholders; clear formulation of objectives; development of a

1 l Mod

ccoriWtFc

fetttnbimitpausmpi

taomss

2

tesorpdr

hracircpt(ic(cmluo

CHM

4 K.A. Rose et al. / Ecologica

onceptual model; documented choices about model approach,omplexity, calibration, verification, and validation; performancef sensitivity analysis and quantification of uncertainties; and peereview and transparency. Jakeman et al. (2006) proposed a sim-lar scheme based on these elements, and Fath et al. (2011) and

ainwright and Mulligan (2004) proposed more generic versionshat use the same basic steps. Espinoza-Tenorio et al. (2012) andAO (2008) also proposed analogous best practices schemes in theontext of modelling for ecosystem-based fisheries management.

The U.S. Army Corps of Engineers (USACE) has put forth a guideor using ecological modeling for ecosystem restoration (Swannackt al., 2012). Once a specific problem has been identified and bothhe planning and modeling objectives have been clearly defined,he basic scheme is: (1) develop a conceptual model identifyinghe specific cause-effect relationships among important compo-ents of the system of interest, (2) quantify these relationshipsased on analysis of the best information possible, which can

nclude scientific data or expert opinion, (3) evaluate the infor-ation yielded by the model in terms of its ability to provide

nformation that describes or emulates system behavior, (4) applyhe model to address questions regarding the effects of particularroject alternatives, and (5) perform periodic post-audits of modelpplications to manage confidence in the model. The guidance doc-ment goes into more detail for each of these steps. They furthertate that “model development iterates through a series of inter-ediate developmental phases (each a more mature form of its

redecessor and sometimes halting further development becausenformation needs are found to have been met).”

Whatever the specific scheme, there are always five componentshat must be undertaken: (1) define the question, (2) organize andssess the data, (3) develop conceptual models, (4) develop a libraryf existing models or develop a new model, and (5) specify theodel and evaluate. These steps are sometimes presented as single

teps in some schemes or are broken down in multiple, sequentialteps in other schemes.

.2. Our proposed scheme

Our version of best modeling practices use various aspects ofhese previously proposed schemes, adjusted and tailored for mod-ling fish responses to large-scale restoration actions. The steps arehown in Fig. 1, and details and suggestions about implementationf each step are described in Appendix A. Below we briefly summa-ize the steps involved. The steps can be categorized into the generalhases of modeling of: planning (steps 1–8), selection (steps 9–16),evelopment (steps 17–24), application (steps 25–28), and publiceporting (steps 29–31).

Steps (1) and (2) involve knowing the details, history, stake-olders, and regulatory aspects of the plan, and how the modelingesults will be used. Step (3) is defining the questions; too littlettention and specificity at this step is the source of most per-eived modeling failures (i.e., unmet expectations). Steps (4) to (7)nvolve developing conceptual models, and step (8) is a contenteview of the conceptual models by regulatory and resource agen-ies (RRA) and by stakeholders. Steps (9) to (11) set-up a library ofossible models and modeling approaches and also a data inven-ory, and these three steps can be done in parallel. Steps (12) to16) are where model selection and specification occurs, includ-ng when financial, technical, and scheduling constraints should beonsidered. There are several exit possibilities during Steps (12) to16) for when the best decision is to conclude that a sufficientlyredible and useful model cannot be achieved. The strategy for

ATTA

odel implementation is documented in Step (17) and then base-ine modeling (verification, calibration, validation, sensitivity andncertainty analyses) is performed in Steps (18) to (22). The resultsf the baseline modeling are presented in a report and are also

elling 300 (2015) 12–29

subjected to a content review of both the model and the baselineresults (Steps 23 and 24). The predicted effects of the restorationactions are simulated, analyzed, and reported in Steps (25) to (27),reviewed by the RRA in Step (28) prior to going public, and thenincorporated into a report (Step 29) that is publically presentedand reviewed (Step 30). The final step (31) is a post-audit of themodeling to ensure it is archived appropriately if the issues are re-visited or if the modeling approach is selected for use in anotherapplication.

There are several places where the steps in our proposedapproach are not simply in a top-to-bottom sequence. The prepa-ration of the strategy document (Step 17) must be cross-checkedwith how the results will be used (Step 2) and the questions to beaddressed (Step 3). This cross-checking is needed because consis-tency is critical and both earlier steps (2 and 3) may have changedfrom their initial formulation to when the strategy document (Step17) is eventually prepared. In addition, the management modelsshould be labelled as such and they play a special role in the strat-egy document because of their previous scrutiny. A commonlyencountered example of a management model is fisheries stockassessments that are updated regularly and used for setting har-vest targets and restrictions; often, these model have undergonetheir own intensive review and scrutiny. Calibration and valida-tion (Steps 20 and 21) almost always require changes to the modeldetermined earlier in Step (16), including updating the strategydocument and repeating model verification. Finally, one of the threeexits (failure to sufficiently calibration or validate—Steps 20 and 21)occurs late in the development process and should be documentedin the post-audit.

The steps involving content review can be challenging to effec-tively navigate. There are multiple types of content review: RRA,peer (P), and stakeholder (S). Interaction with RRA is ongoingthroughout the modeling, but often to a subset of the RRA staff. Peri-odic updates to get feedback from the broader RRA staff can avoidissues being raised near the end and also can help control expec-tations. Peer review focuses on the science (not the policy aspects)and should be at key steps only and in a formal way with presen-tations, written documents, and specific charge questions to theoutside reviewers. Peer review of the model and its analysis shouldbe performed by independent experts, with careful screening forproper expertise and any appearances of conflicts of interest. Stake-holder review is very useful but it is easy for preliminary results tolead to unnecessary angst and inefficient use of everyone’s time. Nomatter how the caveats are stated, they will be dropped from dis-cussions. All three types of review are important and there needsto be a feedback loop from the modelers back to the reviewerson what was changed and what was not changed in response tocomments and why. However, despite these reviews, the modelershave ownership of the model development and use, and the mod-eling cannot be driven by the review groups; the review groups,especially stakeholders, are advisory only.

Effective communication throughout the process is obviouslycritical for the modeling to be properly understood and used indecision-making. A model or modeling analysis should never besimplified, at the sacrifice of realism, in order to make it easier tocommunicate. It is the responsibility of the modelers to effectivelycommunicate the modeling to the RRA, peer reviewers, and stake-holders, but never at the compromise of not using the best sciencein the modeling. There are formal approaches and good templatesavailable for developing conceptual models (e.g., DiGennaro et al.,2012; Fischenich, 2008; Gentile et al., 2001; Ogden et al., 2005) toensure effective communication. Another popular technique is to

ENT 8

develop a simpler model for demonstration purposes; we recom-mend caution in doing this, as it becomes complicated to explainwhen the demonstration model generates different results thanthe production version. Interested parties will select the results

K.A. Rose et al. / Ecological Modelling 300 (2015) 12–29 15

(1) Know th e re stora�on plan

(7) Unify into an ove rall conce ptual model

(12) Candidate models and approaches

(13) Fewer viable models and approac hes

(2) Verify how fish mod eli ng results wil l be use d by R RA

(3) Define the ques �on s to be answered by the modelin g

(5) Popula�on or comm unity dynamics of species of interest

(6) How rest ora� on ac� ons affec t growth, mortali ty, reprod uc� on, and movem ent

(4) Construc t the conceptu al models

(9) Iden� fy exis�ng management mod els

(10) Develop librar y of mod els and approac hes

(11) Create data inv entor y an d summarize prior knowledge

(14) Constraints (fro m RRA )

(15) Even fewer viable models and approaches

(16) Spec ify th e model(s )

(17) Prepare a s trat egy do cument

(19) Perform v erifica�on and di agnos �c tes�n g

(20) Perform cali bra�o n

(21) Perform valida�on

(23) R eport on results for baseline only

(25) S cenarios – FWOA and FWA

(27) Resul ts to RR A

(29) Public Re por�n g

Exit

Exit

(22) Perform sensi�vit y and u ncertain ty analysis

(26) Perfo rm unce rtain ty analysis

(8) Revi ew – R&S

(31) Post-audi�ng

(18) Review – R&P&S

(24) Rev iew – R&P

(28) Review – R

(30) Review – S

Exit

Fig. 1. Flowchart of the steps in our proposed best practices scheme. RRA refers to the regulatory and resource agencies, and peer review is conducted with RRA (R), peerr

ti

aaeau(pham

ATTACHMENT 8

eviewers (P), or stakeholders (S).

hat favor their positions. Visualization can also be very effectiven conveying the dynamics of the model to all audiences.

Finally, a less utilized option is to conclude that existing modelsnd approaches are inadequate, even with modification, to crediblyddress the questions. This occurs when there are no existing mod-ls that sufficiently satisfy the conceptual model, data availability,nd the constraints. The correct action is to then exit and eitherse other methods (e.g., more statistical) or to re-adjust the projecte.g., schedule, budget, expertise) to develop a new model. If one

roceeds with a model that has compromised too much, this willaunt the entire analysis and likely result in rejection of the resultst the end. Exiting is difficult because restoration decisions will beade and it is difficult to determine if the results of a knowingly

weak modeling application are better or worse than no input fromthe modeling at all.

3. Concepts for fish modeling

We also describe 13 concepts that need to be considered inparallel to the steps in our best modeling practices scheme. Theseconcepts, when not considered or not explicitly stated, can derailthe development and use of fish models, even when the best prac-

tices steps are followed. Our experience is that these concepts arenot universally understood and casual treatment of them leadsto mis-communication and mis-understandings, which in turn,often underlie the controversies about the models. We note in

1 l Modelling 300 (2015) 12–29

Ai

3

tldvsBstoEiNt

slidubtssstnSdEsieesn

dqobttam2

TA

CHMENT 8

6 K.A. Rose et al. / Ecologica

ppendix A where one or more these concepts pertain to key stepsn the best practices scheme.

.1. Concept 1: Life cycles and strategies

A life cycle diagram follows individuals as they progress throughhe life stages from birth to death (Caswell, 2001). Developing aife cycle diagram for each species of interest is very helpful wheneveloping a model. A typical fish life cycle is eggs, yolk-sac lar-ae, larvae, young-of-the-year juveniles, juveniles, and adults. Fishpecies often show complicated and diverse life cycles (Able, 2005;alon, 1979; Pihl et al., 2002). Individuals in different stages canhow major changes in their physiology, behavior, diets, and habi-ats utilized, including some stages occurring within estuaries andther stages in coastal and offshore waters (Beck et al., 2001).xamples of well-documented life cycle diagrams can be foundn https://www.dfg.ca.gov/erp/cm list.asp, Baxter et al. (2015), andoble et al. (2009). Such life cycle diagrams clearly show the rela-

ionships among life stages, habitats, and restoration actions.A life history strategy is sometimes confused with a life cycle. A

trategy is determined by the combination of vital rates with theife cycle, and this combination of rates determines how quicklyndividuals go through their life cycle. This is useful for modelevelopment because it provides a way to share parameter val-es among species, for grouping species into functional groups, andecause population responses to perturbations, including restora-ion actions, are expected to be similar among species that haveimilar strategies. Winemiller and Rose (1992) expanded the clas-ical r–K scheme (Pianka, 1970) into a three-end member schemepecifically for fish species (Fig. 2). Fish species fall somewhere onhe surface defined by age or size of maturity, fecundity, and juve-ile survivorship. This model was then expanded by McCann andhuter (1997) to include the salmonids and bioenergetics of repro-uction and growth, and used by Vila-Gispert et al. (2002) to clusteruropean fish species. Able (2005) proposed another categorizationcheme for coastal fish species that used their degree of estuar-ne dependency and connectivity between estuaries and marinenvironments (e.g., residents, breed in marine and juveniles usestuaries), and Pihl et al. (2002) proposed a scheme based on howpecies used habitat within the estuary (e.g., salt marsh for juvenileursery, tidal freshwater for spawning).

Life tables and space/habitat by time plots are helpful for modelevelopment and communication. Life tables provide a means foruantitatively summarizing both the life history cycle and strategyf a species. Life tables show the typical duration and mortality ratesy stage and age. For stages defined by size, life tables also showhe entering weight and lengths, and for adults, they show the frac-

ATTA

ion mature and eggs per individual by stage or age. For example, life table for Atlantic croaker is shown in Table 1, and a review ofethods for constructing life tables is found in Barnthouse (2004,

005). Space-time plots show typical abundances (or biomasses)

able 1n example life table for Atlantic croaker showing vital rates, maturity, weight, and fecun

Life stage Duration (d) Mortality (1/d) Bycatch mortality S

Eggs and larvae 22 0.408 0 0Early juvenile 120 0.00987 0 0Late juvenile 223 0.00987 0 0Age-1 365 0.00123 0.002 0Age-2 365 0.00164 0 0Age-3 365 0.00164 0 0Age-4 365 0.00164 0 0Age-5 365 0.00164 0 0Age-6 365 0.00164 0 0Age-7 365 0.00164 0 0Age-8 and older 365 0.00164 0 0

Fig. 2. Life history strategies for fish species based on age of maturity, juvenilesurvivorship, and fecundity defined by Winemiller and Rose (1992).

by location, season, and life stage, while habitat-time plots showabundance by habitat, season, and stage. These plots add the spa-tial and temporal dynamics to the life cycles, and allow for easydetermination of overlaps among life stages and species (e.g., Able,2005; Pihl et al., 2002).

3.2. Concept 2: Variability, uncertainty, and stochasticity

It is important to distinguish among the sources of variabilityin the model and in the data because they affect the interpretationof data used for calibration and validation, and affect the design ofsensitivity and uncertainty analyses. A convenient classification inmodels used for fisheries management (Harwood and Stokes, 2003)is: (1) process stochasticity, or natural variation, (2) observationerror, (3) model structure errors, and (4) implementation errors.Implementation errors refer to differences between the prescribedrestoration action (often used in the modeling) and the restorationaction when it is actually implemented in nature. The combinedeffects of stochasticity and uncertainty are called variability. Ifmore measurements reduce variability, then one is dealing withuncertainty, whereas when more measurements do not reduce thevariability, one is dealing with stochasticity (Ferson and Ginzburg,1996). Appreciating how variability results from uncertainty andstochasticity sources is important for assessing the fish responsesto restoration actions. Also, a sensitivity or uncertainty analysis

requires specification of how much to vary parameters and whatthe assumed variability means.

Observation error is also important and often ignored. Pre-dicted and observed abundances are compared as part of model

dity by life stage for the first year of life and then by age for age-1 and older.

tage survival Mature (%) Weight (g) Fecundity (eggs/individual)

.00013 0

.306 0

.111 0

.307 0 99.93

.549 50 304.5 465,403

.549 100 562.6 884,453

.549 100 853.2 1356,391

.549 100 1100.1 1757,343

.549 100 1477.8 2370,726

.549 100 1477.8 2370,727

.549 100 1477.8 2370,728

l Modelling 300 (2015) 12–29 17

coihoaoentdo

snbfi(Hgtmita

3

ipOihitcgtllc

3

csydbar(boostbvitbts

Classic Stabi lit y

0255075

100

Bounded Stability

Pop

ulat

ion

Abu

ndan

ce

Episodi c St ability

Regime Shift

Shifting Baseline

Year302520151050

(a)

(b)

(c)

(d)

(e)

MENT 8

K.A. Rose et al. / Ecologica

alibration and validation, and thus the variability around modelutput and the data observations become important to properlynterpret how well the model fits the data. Treating the data asaving no observation error can result in inaccurate determinationf model confidence (Stow et al., 2009). For example, when datare treated as exact or overly precise, the model can be calibratedr validated to differences in the data (e.g., interannual abundancestimates) that are not reflective of variability of the population inature but are due to measurement error. A healthy approach iso view data as also being generated by a model; the model of theata being the sampling frequency, locations, detection limits, andther aspects of the sampling design.

Finally, there is often good reasons and pressure to includetochasticity in model predictions. Purely deterministic models doot capture the true variability observed in nature, and there haseen much effort to incorporate stochasticity and uncertainty intosh population and food web models to match natural variatione.g., Bjørkvoll et al., 2012; Magnusson et al., 2013; Link et al., 2012).owever, it is quite tricky to make a stochastic model that correctlyenerates realistic variability that is comparable to the observa-ional data (Saltelli et al., 2004). Yet, we often interpret stochastic

odel outputs, regardless of how the variability was generated, asf the prediction variability is what is expected in nature. This canhen erroneously influence decisions about whether restorationctions will have detectable and measurable effects.

.3. Concept 3: Generality–precision–realism (Levins)

Levins (1966) proposed that the development of a modelnvolved the trade-offs between generality, realism (accuracy), andrecision. While his idea is still being debated (Evans et al., 2013;rzack, 2005), the concept of trade-offs among these three features

s useful when developing or selecting a model. For example, toave a very general model (highly portable) necessarily means that

t cannot be very realistic AND be very precise (quantitative predic-ions) for a specific location. The key concept is that a single modelannot be formulated that meets arbitrarily defined standards ofenerality, realism, and precision; decisions about model formula-ion requires trade-offs. A highly precise (site-specific) model hasittle precedent for being applied and tested elsewhere, and simi-arly, an off the shelf model (highly general) used in many placesan be criticized for lack of site-specificity.

.4. Concept 4: Nonequilibrium theory, stability, and recruitment

There are very few fish species in large ecosystems that can beonsidered to be in a true equilibrium condition. Equilibrium is apecial form of stability in which abundance is constant year toear. Fish populations, like many animal populations, are not ineterministic equilibrium, but rather show different types of sta-ility (Bjørnstad and Grenfell, 2001; Grimm et al., 1992; Grimmnd Wissel, 1997; Turchin, 2003). Three typical forms are the veryare classical stability where the population goes to a steady stateFig. 3a), bounded stability in which abundances vary year to yearut within a range (Fig. 3b), and episodic stability (Fig. 3c) typicalf many fish species that have highly variable recruitment. Whatften complicates these forms of population stability are regimehifts (deYoung et al., 2008; Scheffer and Carpenter, 2003) that shifthe population up or down at certain times (Fig. 3d), and shiftingaselines (Jackson et al., 2001) that can add a slow trend to theariation (Fig. 3e). Defining baseline conditions is critical becauset provides the basis for then assessing the effects of the restora-

ATTACH

ion actions, but defining baseline in realistic cases (i.e., anythingut deterministic equilibrium) is challenging and getting a modelo then resemble the baseline pattern is frequently only partiallyuccessful.

Fig. 3. Types of equilibrium dynamics common with fish populations.

A major reason for the complicated stability dynamics and defin-ing baseline is the lack of understanding and poor predictability ofannual recruitment of fish populations (Houde, 2008). Recruitmentis defined as the number of individuals surviving to some size or age(e.g., age-1) after which annual mortality becomes less variable. Thehigh and relatively unpredictable variation in recruitment resultsfrom the highly variable nature of survival of young-of-year stagesmakes measurement difficult and because it is the portion of thelife-cycle where density-dependent mortality is assumed to occur(Cowan and Shaw, 2002). A model for assessing restoration willnot solve the long-standing stability and recruitment issues, butone does need to be aware of these concepts and how the modelsselected treat them in order for the modeling to be credible and forthe modeling results to be properly interpreted.

3.5. Concept 5: Scaling

Scaling considers the temporal, spatial, and biological dimen-sions represented in the model. Temporal scale refers to the timestep used in the solution method, the time step at which modelpredictions can be outputted (e.g., daily, seasonally, annually), andthe length of model simulations (e.g., one year or multiple years).Spatial scale refers to the resolution of spatial cells in the model(e.g., 500 m × 500 m) and the domain covered by the model (e.g.,lower portion of the estuary).

Defining the biological scale in a model is more complicatedand involves four major components: (1) currency, (2) organiza-tional level, (3) processes, and (4) prediction level. The currency iswhat is followed in the model calculations as state variables andtheir units. For example, one can simulate total biomass of for-age fish, abundances in age-classes of a single species, biomass

by size classes, or follow individuals. One must also be very clearabout what organizational level is being represented in the model(e.g., single-species, multispecies, community, food web, or ecosys-tem). Multispecies is when some, but not all, species are followed

18 K.A. Rose et al. / Ecological Modelling 300 (2015) 12–29

Fig. 4. The spatial and temporal scales of some of the physical and biological factors that could influence how growth, mortality, reproduction, and movement of fish arerepresented in a model. The solid line ellipses show some of the physical and biological factors, while the dotted ellipses show the fish processes of growth, mortality,r

itCabodm

de2motmss(togatsplfiai

dt

ATTACHMENT 8

eproduction, and movement. Modified from Dickey (2003).

n the model, and thus the sum of biomasses over fish species inhe model does not equal total fish biomass expected in nature.ommunity-level modeling involves following most or all speciesnd thus their summed biomass can be considered total fishiomass. Ecosystem is adding the environment to the multi-speciesr community model. We urge careful use of these terms whenescribing a particular model, as vague references to an ecosystemodel leave too much to the imagination of the audience.Processes are the functions that represent the vital rates that

etermine how individuals progress through their life cycles. Mod-ling of fish involves four basic processes or vital rates (Rose et al.,001): (1) growth, (2) mortality, (3) reproduction, and (4) move-ent. Unlike some other types of modeling, such as hydrodynamics

r statistical, there is no standard methodology for how to representhese processes in fish models. The challenge is illustrated by the

any possible physical and biological factors, and thus the diverseet of spatial and temporal scales, that could be considered. Fig. 4hows the temporal and spatial scales of some key physical factorse.g., turbulence) and biological factors (e.g., plankton phenology)hat could be included as explicit effects on the biological processesf fish. For example, while one would not explicitly include a sin-le variable called “climate” in a fish growth model. Rather, to beble to adequately simulate climate effects would involve adjustinghe inputs to the model to include the effects of ENSO and regimehifts (and associated temperature and circulation patterns), andossibly plankton phenology, zooplankton patchiness, and turbu-

ence (affects foraging). A modeler must decide how to representsh growth, mortality, reproduction, and movement in their model,nd what environmental and biological factors to make explicit or

mplicit, or ignore.

The fish processes are also interrelated. Mortality and repro-uction affect the numbers of individuals alive at any givenime; growth affects body size and thus affects biomass

(number × weight) and can indirectly affect mortality and repro-duction because these are often related to body size (Rose et al.,2001). Movement determines where individuals are located, andthe environmental (e.g., temperature) and ecological conditions(e.g., prey, competitors, predators) they experience. These then canaffect growth, mortality, and reproduction. While the movement ofeggs is dominated by physics (i.e., transport), juveniles and adults,and even larvae, can exert significant control via behavior over howthey move that then affects their process rates (Jørgensen et al.,2013; Lett et al., 2009). How movement is represented must beclearly described in all spatially-explicit models.

Prediction level refers to the output of the model and how modelresults should be interpreted, and thus how model results shouldbe compared across different simulations. Typically in evaluation ofrestoration actions, predictions of abundance and biomass are com-pared between a status quo (baseline or no-action) simulation anda simulation that includes the effects of restoration projects. Thus,most fish models are best used to predict changes in abundanceand biomass, but yet can easily be mis-interpreted as predictors ofabsolute abundances and biomass because of the labelling of themodel output as total biomass or abundance versus actual years.In fisheries modeling, there is often a mix of absolute and relativepredictions reported within the same analysis (e.g., Kaplan et al.,2012). Interpretation and stake holder expectations should be care-fully considered when presenting model predictions, with cleardistinction between forecasting (sensu Clark et al., 2001), long-term average predictions of absolute quantities, and prediction ofrelative changes.

3.6. Concept 6: Explicit versus implicit representation

The linkages between the effects of the restoration actionson environmental conditions – via changes in hydrology, water

l Mod

qvn(tbiaahli

ttelttcta1rmIs

3

ldfefwohppdsfir1

3

mtdgmodpatiaaKtl

MEN

K.A. Rose et al. / Ecologica

uality, and habitat – and the fish responses in their process andital rates can be explicitly or implicitly represented. A model doesot necessarily require a variable labeled with the restoration effecte.g., more or less wetlands) in order to include the effects of addi-ional or reduced wetlands on fish in a model, and similarly, simplyecause a model has a variable labelled “wetlands” does not mean

t can just be changed to simulate the effects of the restorationction. What are important are not the existing or missing labels of

model, but rather the actual equations. While documentation iselpful, often the only true way to see exactly the process formu-

ations, and therefore how restoration effects can be represented,s by looking at the computer source code itself.

Implicit versus explicit representation also applies to spatial andemporal considerations. One does not have to simulate the spa-ial and temporal scales of every process in order to include theirffects in simulations. For example, prey encounters occur on mil-imeters and second scales, but one does not have to build a modelhat uses millimeter-sized spatial cells and a one second time stepo include predators’ encountering patchily distributed prey. Wean do it implicitly by generating randomness around the func-ion that relates prey to predator consumption or growth (Letchernd Rice, 1997; Pitchford and Brindley, 2001; Winemiller and Rose,993). However, there are limitations to the implicit approach toepresenting variability in that it becomes very difficult to haveemory from one time to the next time (DeAngelis and Rose, 1992).

mplicit representations are better at adding variability that is nottate dependent (i.e., independent from time step to time step).

.7. Concept 7: Population definition

Many of the fish species of interest in restoration show complexife cycles that involve individuals that exit and enter the modelomain and when outside the domain, mix with other individualsrom other regions (see Hixon et al., 2002). For many species, mod-ls will be used for multiyear simulations that include reproductionrom modeled individuals as if the population was self-containedithin the modelled region (i.e., closed population). Dealing with

pen or partially-open populations requires care with regard toow the model predictions are interpreted. For example, sayingredictions over multiple years from the model are long-termopulation responses can be incorrect. Individuals can experienceifferential survival, growth, and reproduction while they are out-ide the model domain and many species do not show high sitedelity and so assuming all individuals that leave subsequentlyeturn is often not valid (Cowen and Sponaugle, 2009; Hughes,990; Jones, 2006).

.8. Concept 8: Density-dependence

Density-dependence refers to how the vital rates (growth,ortality, reproduction, and movement) change in response to

he number of fish individuals present. Compensatory density-ependence is when high numbers of individuals cause slowedrowth, higher per capita mortality, reduced reproduction, orovement to less optimal habitat (i.e., where mortality is higher,

r growth or reproduction is lower). Compensatory density-ependence is a negative feedback and acts to stabilize theopulation (i.e., leads to a dampening of interannual variation)nd also dampen the responses to the positive effects of restora-ion actions. Depensatory density-dependence is when mortalityncreases or reproduction decreases at low numbers of individu-ls, and is a positive feedback and therefore a destabilizing factor

ATTACH

t low abundances (Liermann and Hilborn, 1997; Walters anditchell, 2001). Depensatory density-dependence can be impor-

ant for restoration because targeted populations are often atow abundances when depensation occurs. The difficulty is that,

elling 300 (2015) 12–29 19

while it is known that compensatory and depensatory density-dependence exist and operate, it has been difficult to quantify themwith high precision (Rose et al., 2001; Sable and Rose, 2010). Acommon way to quantify density-dependence in fish models isthrough a spawner-recruit relationship in which recruitment lev-els off (Beverton–Holt) or even begins to decrease (Ricker) at highspawning biomasses (Cushing, 1996). It is possible to overesti-mate responses to restoration actions if density-independence isassumed because the density-dependence that occurs in nature willact to dampen the response.

3.9. Concept 9: Verification, calibration, and validation

Verification, calibration, and validation are three very importantsteps in establishing model credibility. Swannack et al. (2012) offeruseful definitions:

Calibration: The process of adjusting model parameters withinphysically defensible and ecologically reasonable ranges, until theresulting predictions give the best possible fit to the observed data.In some disciplines, calibration is also referred to as “parameterestimation.”

Verification: Examination of the algorithms and numerical tech-niques in the model to ascertain that they truly represent theconceptual model, and that there are no inherent numerical prob-lems with obtaining a solution. In some disciplines, verification isalso referred to as “code testing.”

Validation: The process of confirming a model’s applicability,usually conducted by applying a calibrated model to a set of dataseparate from that used in the calibration process, to demonstratethe accuracy of predicted results. In some disciplines, validation isalso referred to as model evaluation or skill assessment.

Verification is often not documented and is underappreciated.Software testing is required to ensure the model solution is correctwithin some known, sufficiently high, precision. Verification is alsoneeded to ensure that the actual coded equations are consistentwith the conceptual model. Much discussion will take place aboutthe model using the conceptual model as a surrogate for the codedequations so their exact correspondence is necessary.

Calibration is needed because parameter values are taken frommultiple sources that involve other species, systems, time peri-ods, or laboratory conditions. Thus, when all the values are simplyplugged in, it is quite reasonable that some additional adjustmentwill be needed to have all the parameter values fit together andresult in realistic model behavior. Calibration can vary from adhoc adjustment by the modeler until desired qualitative behav-ior occurs (e.g., equilibrium) to formal optimization being used todetermine the parameter values that minimize the differences (e.g.,nonlinear least-squares) between predicted values and observeddata (Janssen and Heuberger, 1995; Rose et al., 2007; Wu and Liu,2012). How calibration is performed needs to be clearly stated.

Validation is a more elusive goal than calibration; yet, it is whatmany people immediately look at to determine if they “believe”the model results. Even what is meant by “validation” has beendebated and discussed for decades (Barlas and Carpenter, 1990;Refsgaard and Henriksen, 2004; Rykiel, 1996). Wainwright andMulligan (2004) offer a good list of ways to validate models(Table 2). With the rise of individual-based modeling (DeAngelisand Mooij, 2005), the idea of pattern matching (Grimm et al., 2005)over goodness of fit statistics based on predicted versus observedhas also gained traction.

3.10. Concept 10: Sensitivity and uncertainty analysis

T 8

Sensitivity analysis and uncertainty analysis are used to estab-lish the robustness of model results and to identify those para-meters that greatly influence model behavior. Sensitivity analysis

20 K.A. Rose et al. / Ecological Modelling 300 (2015) 12–29

Table 2Types of model validation methods from the list proposed by Wainwright and Mulligan (2004).

Type Method

Face validation The evaluation of whether model logic and outputs appear reasonableTuring tests “Experts” are asked to distinguish between real-world and model outputVisualization techniques Often associated with a statement that declares how well the modeled results match the observed dataComparison with other models Similarity of model predictions to the predictions from other models. (note the high likelihood of developing an

argument based on circular logic here)Internal validity Using the same dataset repeatedly in a stochastic model to evaluate whether the distribution of outcomes is always

reasonableEvent validity Whether the occurrence and pattern of a specific event are reproduced by the modelHistorical data validation Using split-sample techniques to provide a subset of data to build a model and a second subset against which to test

the model resultsExtreme-condition tests Whether the model behaves “reasonably” under extreme combinations of inputsTraces Whether the changes of a variable through time in the model are realisticSensitivity analyses Evaluate whether changes in parameter values produce “reasonable” changes in model output

, and

ith act mode

uau2uaass

uviwtsu

3

emm(Dsdi2stqvCttt

awfoittao

ATTACHMENT 8

Multistage Corresponding to the stages a, bPredictive Comparison of model output wStatistical Indicating whether the range of

ses small changes in parameter values, usually applied equallycross all parameters (e.g., ±10%), whereas uncertainty analysisses realistic variation in parameter values (Saltelli and Annoni,010). Monte Carlo methods can be used for sensitivity analysis,ncertainty analysis, and for representing variability (stochasticitynd uncertainty) in stochastic models. The key is how much inputsre varied and whether they are varied only at the beginning of aimulation (sensitivity and uncertainty analyses) or throughout aimulation (stochastic model).

In many restoration situations, outputs from other models aresed as inputs to the fish models. Often, averaged or snapshotalues are passed from one model to another without accompany-ng variability. Treating outputs of other models as known values

ithout variability can distort the results from the fish modelshat receive these as inputs. Thus, the propagation of variability,tochasticity, and uncertainty through coupled models is of partic-lar concern.

.11. Concept 11: Multiple modeling strategies

There are three common situations that involve multiple mod-ls: ensemble modeling, dueling modeling, and as part of coupledodel analysis. Ensemble modeling uses the results of multipleodels to deal with uncertainty due to model structural errors

i.e., agreement implies robust results) (e.g., Gårdmark et al., 2013).ueling models occurs when people (e.g., stakeholders) are not

atisfied with the results of a model and so attack it, and oftenevelop their own model, and the result can be multiple models

n a confrontational situation (e.g., Barnthouse et al., 1984; NRC,010; Swartzman, 1996). Critical aspects of the multiple modelituations are understanding the history of the models; the motiva-ions underlying their development; and careful evaluation of theuestions addressed, process representations, parameter and inputalues, and spatial and temporal aspects of their implementations.lear explanation of multiple models and how independent theyruly are is often lacking or fuzzy, which creates confusion amonghe various participants and stakeholders about how to interprethe multiple model results.

Coupled modeling is often not well documented and described,nd often ignores the uncertainties in the donor model. Describinghether the coupled models are solved simultaneously (allowing

or two-way exchange) or semiautonomously by using storedutput from the donor model is very important. Another aspects how the output of the donor is aggregated (e.g., averaged over

ime and space) to fit into the structure of the fish model. Both ofhese issues also apply when the fish model is the donor modelnd its output is used as input to other models (e.g., managementr economic models).

c noted aboveual behavior of the system in questionl behavior and its error structure match that of the observed system

3.12. Concept 12: Food web dynamics

Multi-species considerations and food web dynamics are oftenconsidered the best level at which to evaluate restoration actions,but are challenging to formalize into mathematical and simula-tion models. It is relatively easy to develop a conceptual modelof a simple or even complex food web and people know iso-lated species do not exist in nature; thus, the idea of multispeciesor food web modeling seems reasonable (Rose and Sable, 2009).However, translating the arrows showing food web interactions(predator–prey) into a numerical version is a major challenge(Chiu, 2013; Pinnegar et al., 2008). Food web modeling fur-ther makes model development subject to developer judgment,and yet single-species approaches are criticized for ignoring thefood web interactions (Hilborn, 2011). The debate over prey-dependent versus ratio-dependent predation (DeAngelis, 2012)and the important role of weak interactions within food webs(McCann et al., 1998) are two examples of issues that makefood web modeling challenging. Thus, food web or communitylevel modeling for restoration analyses should proceed, but withparticular attention to treatment of the major uncertainties sur-rounding representation of inter-specific interactions.

3.13. Concept 13: Hidden assumptions and domain ofapplicability

Fish models have many hidden assumptions. A major hiddenassumption is about the range of input values over which cer-tain relationships are valid and the labeling of inputs with generalnames but then using them in very specific ways. These hiddenassumptions, plus the range of conditions encompassed by the cal-ibration and validation, define the domain of applicability of themodel. The domain is the range that the driving variables andparameters can be changed, while still remaining confident of themodel’s realism. To illustrate, consider a model that has an inputlabeled “salinity.” However, the equation in the model that usessalinity was a linear relationship of its effect on growth, and esti-mated over a narrow range of values. Also, because of the previouslysmall variation in salinity, the effects of salinity on other possibleeffects (e.g., movement) were ignored. Thus, simply changing salin-ity in the model can result in inaccurate predictions of the effect ofthe restoration action. Inaccuracies arise in this example becausethe new values of salinity actually cause more than linear changein growth that is assumed in the model, and because the fish would

no longer be spatially distributed the same and thus are simulatedto experience incorrect temperatures and food conditions.

Other hidden assumptions are disconnects between writtendocumentation and what the code actually does and the solution

l Mod

mdbnabcub

4

sfssa

4

fpmaascmetCrta

tb(wsSedhWiwtoohteia(lmoa

hl

MEN

K.A. Rose et al. / Ecologica

ethods used. A published paper on the model may have oneescription or lack details, and the only way to know what is reallyeing computed is to look at the source code. The solution methodeeds to be considered because the order of solving things canffect results, and the numerical aspects of the solution need toe confirmed because changes in parameter values or equationsan change the numerical accuracy of solutions. All models havenderlying mathematics and their solution must be confirmed aseing sufficiently precise.

. Case studies

We use three case studies to illustrate how several of the keyteps and their associated concepts are used in fish modelingor restoration. Our examples provide some concreteness to thecheme and evidence that our steps are not individually a hugetep away from common practices. The key is to put all of the stepsnd concepts together.

.1. Case study 1: Colorado River and Glen Canyon Dam

Glen Canyon Dam on the Colorado River immediately upstreamrom the Grand Canyon caused significant disruption to native fishopulations downstream by altering the flows of water and sedi-ent. The dam, closed in 1963, impounds Lake Powell that contains

bout 35.5 km3 of water, which is equivalent to almost 3 year’sverage flow of the river. The dam releases water drawn from deeptrata of the reservoir where water temperatures are substantiallyooler than the natural flow of the river. The reservoir also storesost of the 6 × 107–1.2 × 108 kg of sediment entering its reservoir

ach year (Graf et al., 2010). This storage of sediment results inhe replacement of naturally turbid waters downstream in Grandanyon with releases of relatively clear water. The dam essentiallyemoved large flood events from the downstream river regime, sohat the aquatic and riparian habitats of the river, once connectednd dynamic, became disconnected and inactive.

Fish populations below the dam responded dramatically tohese bio-physical changes caused by the dam closure. Humpedack chub (Gila elegans Baird and Girard) and razorback suckerXyrauchen texanus Abbott) were attuned to warm, sediment-ladenaters and have declined in numbers (Minckley, 1991). These two

pecies are now listed as endangered under the U.S. Endangeredpecies Act, while several other species have been extirpated or arexceedingly rare. The segment of the river extending about 24 kmownstream from the dam has relatively clear, cold water, andas become a globally recognized rainbow (Oncorhynchus mykissalbaum) trout fishery. A large release of water from the dam dur-

ng a period characterized by high precipitation in the upstreamatershed in 1983 coincided with widespread recognition that

he operation of Glen Canyon Dam was having deleterious effectsn the downstream ecosystem and its fishes. When the dam’sperators, the U.S. Bureau of Reclamation, sought to upgrade theydropower turbines in the dam and to change operating rules,he required environmental impact assessment lead the Bureau tostablish the Glen Canyon Environmental Studies unit in 1986 tomprove understanding of the river, its ecosystem, and the man-gement options that might be employed to restore the systemNRC, 1987). Experimental flood releases from the dam have simu-ated minor natural floods without compromising the overall water

anagement objectives of the dam that are tied to a lengthy historyf legal agreements (Summitt, 2013), and native fish populations

ATTACH

re increasing.We now enjoy the perspective of more than 30 years of project

istory in a search for lessons learned for other projects thatack such a lengthy history. The Glen Canyon–Grand Canyon case

elling 300 (2015) 12–29 21

illustrates three important components in the best practices wepropose in this paper: peer review (multiple steps), defining thescale of analysis in a conceptual model (step 4), and the creation ofa data inventory and synthesis of knowledge (step 11).

Peer review of the resulting restoration plans and operationsbegan in 1986 with the appointment of a series of National ResearchCouncil committees whose activities extended over a decade. Sub-sequently, the establishment of a monitoring and research centerfor restoration of the system included a peer review team that hascontinued to work closely with researchers and managers withconsiderable success.

The spatial scale of analysis for fish models was fairly clearfrom the beginning in the Glen Canyon case (part of step 4). Theriver extending 446 km downstream from the dam through GrandCanyon and terminating at the headwaters of Lake Mead, was thelogical geographic domain. However, the appropriate boundariesfor water and sediment that controlled ecosystem conditions forfish was a different matter. Reservoir operations in Lake Pow-ell strongly affect downstream flows, so the upstream extent forhydrologic models had to include the lake. Sediment in the channelbelow the dam comes from two primary tributaries, the Little Col-orado River from the south and the Paria River from the north, andadditionally from flood bars stored along the main river. Sedimentmodeling came to include all these sources at a somewhat differentspatial scale than the fish models. Connecting and coordinating themodels was a surmountable challenge.

The creation of a conceptual model (step 4 of our best practices)was a key to successful ecosystem restoration. This conceptualmodel evolved over a period of years as a product of the interactionbetween agency personnel and their peer reviewers. Many peoplecollaborated to create a complicated picture of the project compo-nents and their connections in the form of a diagram. Although theso-called “spaghetti diagram” was the subject of more than a fewjokes, it was also a valuable tool in visualizing the system oper-ation, serving as a decision support tool, helping define researchproblems, and facilitating communication (NRC, 1996).

Data inventory (step 11) has been a critical component of therestoration of the fish populations in the Colorado River of theGrand Canyon. The 1992 Grand Canyon Protection Act directedthe Department of Interior to create a long-term monitoring andresearch program to support adaptive management (USGS, 2011).The Grand Canyon Monitoring and Research Center was createdto serve as a data storage and retrieval center with advice fromperiodic peer review. The center serves to integrate geophysical,hydrologic, and biological data with bearing on the Colorado Riverfishes. The Colorado River case is more mature than some otherexamples, partly because it has been in existence for three decades,but it can serve as a model for other peer-review efforts for fishmanagement in restoration projects.

4.2. Case study 2: Restoring coastal Louisiana

Coastal Louisiana has lost 4877 km2 of wetlands since the 1930sand is projected to lose about 4500 km2 over the next 50 years(Peyronnin et al., 2013). Historically, these losses have comprisedabout 90% of the national loss of coastal wetlands (Caffey andSchexnayder, 2002). The losses result from channelization of theMississippi River preventing land building from periodic flooding,and from subsidence and sea level rise. Coastal wetlands providestorm surge protection, nutrient processing, and nursery habitat formany coastal fish species (Barbier et al., 2011; Engle, 2011). As partof planning for the 5-year update to the 2012 state-wide restora-

T 8

tion plan, a strategy for selecting fish models to assess restorationeffects was implemented (Rose and Sable, 2013). The 2012 planrelied on habitat suitability analyses, and the idea was how to selectfish models beyond HSI for the 2017 update. This strategy follows

22 K.A. Rose et al. / Ecological Modelling 300 (2015) 12–29

Table 3A five category scheme used to summarize the initial list of available models for coastal Louisiana. Multispecies includes a few species (<5), compared to community-level,and ecosystem is when environmental variables are added. Models can start with a specified number of starting young (i.e., forced recruitment), or the young can be produceddynamically based on the model simulated adults (i.e., full life cycle). Multiyear runs can be either forced recruitment or full life cycle.

Biological Spatial Temporal Reproduction

Currency Organization

State variable Single-species Point Seasonal Forced recruitmentAge structured Multispecies Spatially explicit One year Full life cycleStage structured Community Multiple years

abtt

om

1

2

3

re(irollatb

iomwbheettrfirmsetrc

ATTACHMENT 8

Individual-based Food webEcosystem

nd illustrates steps 3 (questions), 10, and 12 through 15 of ourest practices scheme. The result was a well-documented and jus-ifiable decision on what fish models to use for the next version ofhe coastal restoration master plan.

After several meetings with the state agency personnel in chargef coastal restoration, the questions to be answered by the fishodels (Step 3) were defined as:

. How does each project affect the distribution and relative abun-dance (i.e., biomass or density) of each key species versus theirpredicted abundance in the Future without Action condition overa 50 year period?

. How do select sequences/combinations of projects withineach coastal basin affect the distribution and relative abundanceof each of the key species versus their predicted abundances inthe Future without Action condition over a 20 year period?

. How does the coastwide restoration/protection plan affect thedistribution and relative abundance of each of the key speciesversus their predicted abundance in the Future without Actioncondition over a 50 year period?

Recognition that a model could not deal with the fine tempo-al and spatial scales of question 1, and yet be able to be broadnough for questions 2 and 3, lead to a focus on questions 2coastal basin scale) and 3 (coastwide). The same meetings resultedn a list of fish species of interest (i.e., the key species), and theeasons why they were selected. Reasons included one or moref the following: commercial or recreational importance; eco-ogical importance; controversial relevance; expected to exhibitarge responses; and influenced by relatively expensive restorationctions. Once selected, key species were viewed in life history spaceo confirm that the major life history types that would be affectedy the various restoration actions were represented.

More specificity in the questions (Step 3) was achieved by defin-ng the term “effects”, by eliminating from consideration somef the restoration actions, and by clearly stating the form of theodel outputs that was needed. Several of the restoration actionsere eliminated from the fish modeling analyses (e.g., bank sta-

ilization) because they had resulted in only small changes inabitat suitability in earlier analyses, and would require fish mod-ls to work on very fine spatial and temporal scales. We definedffects, as used in the modeling questions, as the series of stepshat link changes in the environment caused by the restora-ion action (e.g., marsh creation) to changes in growth, mortality,eproduction, and movement rates and patterns of the individualsh. Finally, it was determined that consistency in the predictedesponses from the models across species was more important thanechanistic understanding of a particular response. Less empha-

is on mechanistic understanding allows for flexibility to use more

mpirically-based relationships for growth, mortality, reproduc-ion, and movement, as long they could accommodate the expectedange of changes in environmental conditions (i.e., domain of appli-ability).

The Louisiana study also illustrates how one goes from a modellibrary (Step 10) to the selected models and approaches (Step 15).A model library was developed (Step 10) with two levels: an ini-tial listing of about 30 possible models and further refined list ofcandidate modeling approaches from these 30 that survived an ini-tial screening. Some of the models reviewed were general softwarepackages (e.g., Ecopath with Ecosim [EwE]); for those that had mul-tiple versions available, we summarized specific implementationsin the following order of priority: (1) Louisiana version, (2) Gulf ofMexico version, (3) version that simulated a key habitat (e.g., wet-lands), and (4) recent implementations. For the initial listing of themodel library, a five-category taxonomy based on currency, biologi-cal organization, spatial and temporal scales, and how reproductionwas represented (Table 3) was used. A portion of the resulting initiallisting of the 30 models is shown in Table 4.

Next, nine fish-oriented models were selected from the initiallist (Step 13) that appeared to match, or could be modified to match,the conceptual model. The reasons for eliminating the other modelswere also explained (Table 4). The nine models selected for fur-ther evaluation were summarized in much more detail based on:level of complexity (e.g., number of species), spatial representa-tion (resolution and domain), temporal aspects (time step, single ormore years), fish processes included, availability of information oncritical parameters and environmental inputs, form of the modeloutputs, mathematics and computing details, and availability ofsource code. At this time, these features were also viewed for theirrelevance to the specifics of the anticipated 2017 Coastal MasterPlan actions and ecological questions. For example, did the tempo-ral and spatial features allow for 50-year simulations at the scalesof the basin and coastwide? This listing became the basis for even-tually selecting modeling approaches to be implemented (Step 15),including one of the final steps of applying the constraints (Step 14)to the nine possible approaches.

The constraints of Step (14) were defined to the extent possi-ble in this early planning stage. Constraints related to schedulingincluded the start dates and milestone dates of key intermediateand final products. These were determined by starting with thefinal reporting and working backward. Preliminary estimates oftotal costs (financial constraints) were determined based on pastexperience of the modelers involved, and it was deemed neces-sary that the modelers work with the source code of all modelsselected. An important technical constraint was that, at a mini-mum, any selected fish models need to be capable of being coupledto the other models in semiautonomous and one-way modes. Toassess this, the outputs of the other models that would becomeinputs to the fish models were summarized in detail in terms oftheir spatial and temporal scales, and how they could be averagedand aggregated to accurately be passed as inputs to the fish models.Too often in other situations, vague statements are made about how

models are coupled; specificity is important. It was also noted thatthe other component models all exchanged information in a two-way coupling, and that there are some situations where two-waycoupling could be important with the fish models. For example, if

K.A. Rose et al. / Ecological ModTa

ble

4Ex

amp

le

list

ing

of

the

init

ial m

odel

s

con

sid

ered

for

the

Lou

isia

na

Mas

ter

pla

n. S

ix

are

show

n

from

a

tota

l of m

ore

than

30.

Mod

el

Loca

tion

Hab

itat

Cu

rren

cy

Bio

logi

cal

orga

niz

atio

nSp

atia

l

Tem

por

al

Rep

rod

uct

ion

Lotk

a-V

olte

rraa

Non

e

Not

spec

ific

Stat

e

vari

able

Mu

ltis

pec

ies

Poin

t

Mu

ltip

le

year

s

Imp

lici

tly

rep

rese

nte

dR

easo

n

elim

inat

ed:

mod

el

is

hig

hly

aggr

egat

ed

and

doe

s

not

allo

w

for

suffi

cien

t

real

ism

nor

the

rep

rese

nta

tion

of

the

man

y

effe

cts

to

hyd

rolo

gy

and

wat

er

qual

ity

Stri

ped

mu

llet

stoc

k

asse

ssm

entb

Lou

isia

na

Coa

stw

ide

Age

-str

uct

ure

d

Sin

gle

spec

ies

Poin

t

Mu

ltip

le

year

s

Full

life

cycl

eR

easo

n

elim

inat

ed:

a

stat

isti

cal c

atch

-at-

age

mod

el

doe

s

not

per

mit

easy

sim

ula

tion

of

the

effe

cts

on

hyd

rolo

gy

and

wat

er

qual

ity

EwEc

Bre

ton

Sou

nd

, Lou

isia

na

Estu

ary

Age

-str

uct

ure

d

for

som

e;

Stat

eva

riab

le

for

oth

ers

Ecos

yste

m

Poin

t

Mu

ltip

le

year

s

Forc

ed

recr

uit

men

t

Rea

son

elim

inat

ed:

EwE

was

not

elim

inat

ed

and

was

use

d

to

illu

stra

te

an

app

roac

h;

Gu

lf

of

Mex

ico

and

Lou

isia

na

vers

ion

s

exis

tC

ASM

dPo

ntc

har

trai

n

Bas

in, L

ouis

ian

aEs

tuar

y

Stat

e

vari

able

sEc

osys

tem

Poin

t

1989

-200

7 Fo

rced

recr

uit

men

tR

easo

n

elim

inat

ed:

CA

SM

was

not

elim

inat

ed

and

was

use

d

to

illu

stra

te

an

app

roac

h;

Lou

isia

na

vers

ion

s

exis

tSh

rim

p

IBM

eLo

uis

ian

a

and

Texa

s

mar

shes

Veg

etat

ion

and

open

wat

erIn

div

idu

al

juve

nil

ebr

own

shri

mp

Sin

gle

spec

ies

Spat

iall

y

exp

lici

t

On

e ye

ar

Rea

son

elim

inat

ed:

not

elim

inat

ed

and

use

d

to

illu

stra

te

an

app

roac

h. H

owev

er, t

he

fin

e

spat

ial r

esol

uti

on

(met

ers)

lim

its

the

geog

rap

hic

scal

e

of

mod

el

pre

dic

tion

sSp

atia

l mu

ltis

tock

pro

du

ctio

nm

odel

fB

isca

yne

Bay

, Flo

rid

aC

oral

, sea

gras

s,h

ard

and

soft

bott

om

Age

-str

uct

ure

d

Mu

ltis

pec

ies

Spat

iall

y

exp

lici

tO

ne

year

Full

life

cycl

e

Rea

son

elim

inat

ed:

not

elim

inat

ed

and

use

d

to

illu

stra

te

an

app

roac

h

aW

hip

ple

et

al. (

2000

).b

Wes

t

et

al. (

2013

).c

de

Mu

tser

t

et

al. (

2012

).d

Bar

tell

et

al. (

2010

).e

Rot

h

et

al. (

2008

).f

Au

lt

et

al. (

1999

).

ATTACHMEN

elling 300 (2015) 12–29 23

the responses of Gulf menhaden (Brevoortia patronus) were simu-lated in a fish model, then the consumption of phytoplankton bymenhaden may be important enough that the two models must betwo-way linked in order to get realistic simultaneous predictions ofchlorophyll and menhaden. Finally, the technical constraint of com-puting speed was considered but was unable to be assessed in detailbeyond acknowledging that development, calibration, validation,and scenarios analyses of any selected fish models would require100s of 50-year simulations over roughly an 18 month project timeperiod. Thus, any preconceived or intuitive ideas of limitations dueto computing were not prematurely used to eliminate or favor cer-tain models and approaches. The end result was a final list of 9 fishmodels and approaches (Step 15), of which a subset are presentlybeing considered and implemented (Step 16) for possible use in the2017 master plan.

4.3. Case study 3: Florida everglades

The Florida Everglades is a subtropical, seasonally-pulsed,karstic marsh that evolved a complex topographic braided ridge-and-slough structure (Larsen et al., 2007). Variation in topographicrelief in these peat ridges creates a variety of shallow water habi-tats of different water depths that are exploited by many speciesof small fish and invertebrates. The Everglades is a nutrient-limitedsystem, but small fishes (less than 8 cm) transmit sufficient biomassto support higher trophic levels. Small fish provide prey to uppertrophic levels by their movement across the landscape and theirrapid growth that is linked to the seasonal hydrology. The floodedarea of the Everglades greatly expands during the wet season,allowing populations of small fishes to disperse across the marshand increase in body mass and total biomass. During the dry season,as water levels fall, the biomass of the community of small fishesis then concentrated in pools and depressions and becomes avail-able for wading birds. But the decrease in the length of floodingperiod (hydroperiod) in recent decades due to water managementhas diminished the ability of fish biomass to build up to high levelsand to be concentrated at critical times for apex predators. Impactson fish and invertebrates communities, and the top predators thatfeed on them, have been severe.

Better understanding and prediction of fish production in theEverglades was needed with the 1997 Congressional enactmentof the Central and South Florida Comprehensive Review Study(Restudy) to develop a long-term plan for the Everglades restora-tion. Because fish provide much of the food base for wading birds,alligators, and many other animals, it was essential to assess theirresponse to changes in hydrology before expensive changes in thesystem of water regulation were made, and modeling was pro-posed as one of the tools for such assessment. There was muchdiscussion and several workshops (e.g., Science Sub Group, 1996)surrounding the definition of the questions to be addressed andhow modeling could address various aspects of the questions(Step 2).

Characterization of the fish populations across the Evergladeslandscape required combining ecological and hydrological mod-els along with empirical data on hydrology and fish populations.The hydrology of this area is complex, with water depths chang-ing seasonally and drying out completely in some areas in manyyears. The fish populations undergo local dynamics, but they canalso move in response to changing water levels, tending to retreatinto deeper water when water levels go down and to expandacross the marsh when the marsh floods. To simulate this highlydynamic ecosystem component, a computer landscape model for

T 8

the fish functional groups, ALFISH, was developed (DeAngelis et al.,1997, 1998) as part of the U. S. Geological Survey’s Across TrophicLevel System Simulation (ATLSS) project. ALFISH was used toproduce spatially-explicit estimates of the fish number density

2 l Mod

asmertdsb(

Amtscoggbowsfpoln

A2t(sshtsfrmtifito

mcmfidwldptacAcoa2

i

CHM

4 K.A. Rose et al. / Ecologica

nd biomass prey base across the greater Everglades, over theame area that is covered by the South Florida Water Manage-ent Model (Fennema et al., 1994). ALFISH had the objective of

valuating the small fish response to proposed alternative wateregulation scenarios designed to restore some of the key fea-ures of hydrology of the Everglades. The ALFISH model wasesigned to have the capability of providing a dynamic mea-ure of the spatially-explicit food resources available to wadingirds. This defined the specific questions the model would addressStep 3).

A conceptual model was first developed (Step 4) based on anTLSS workshop in 1996, and a review of the literature on fishodeling (Step 10) was prepared (Gaff et al., 1999). The concep-

ual model consisted of two functional groups of fish (large andmall), where the larger fish could prey on the smaller. The con-eptual model documented how the fish populations could spreadut spatially in response to flooding during the rainy season androw in population size and either retreat into refugia (e.g., alli-ator ponds), move to areas that were still flooded, or die if theyecame stranded. Of special importance was the creation of pocketsf high fish concentration during the drydown periods, where theyould be available to foraging wading birds during their nesting

eason. The fish populations had to be simulated across the wholereshwater Everglades, over a 30 year time period, and at the tem-oral resolution of a few days. It was decided that no existing modelf fish dynamics was appropriate for describing fish on a dynamicandscape such as the Everglades, and thus a new model wouldeed to be developed (Step 15).

The development, testing, application, and reviewing of theLFISH model followed, to various degrees, many of Steps 9 through4, including documentation (Step 17), verification and diagnos-ic testing (Step 19), calibration (Step 20), and sensitivity analysisStep 22). To implement the conceptual model, the marsh land-cape was modeled as 500 × 500 m spatial cells on a grid acrossouthern Florida. The South Florida Water Management Modelydrology model predicted water levels in the spatial cells on 5-dayime steps. Procedures detailed in DeAngelis et al. (1998) allowedpatially detailed maps of estimated topography to be developedor large wetland regions with low topographic relief. The topog-aphy was then coupled to the hydrology data to provide dynamicaps of water depth on daily time steps and a 500 × 500 m resolu-

ion (Sylvester, 1998). Finally, this high resolution hydrology wasntegrated and averaged for every 5 days to create the specific datale needed as inputs to ALFISH. ALFISH was then used to determinehe weekly and seasonal pattern of fish densities on the landscapever multiple decades.

ALFISH was then used to simulate a series of alternative wateranagement and restoration strategies. Model predictions were

ompared with two alternative baseline cases in order to deter-ine the benefits and problems with each scenario (Step 25). The

rst baseline was based on a 31-year time series of historical rainfallata from 1965 to 1995 with other conditions set to 1995 values,hile the second baseline used the same rainfall data but with sea

evel, population level, and socioeconomic conditions set to pre-icted values for year 2050. Maps showing the model results wereroduced for each of two baselines, a management alternative, andheir differences. Results of these comparisons were summarizednd submitted for review to aid in making improvements in suc-essive scenarios (Step 27), as reviewed by Gaff et al. (2000). TheLFISH model was extensively reviewed by the responsible agen-ies involved in Everglades restoration, as well as by a team ofutside reviewers (Gardner et al., 2002) in 2002 (Step 30), and key

ATTA

ssumptions were evaluated using sensitivity analysis (Gaff et al.,004, Step 22).

In order to make more specific predictions of the effects of vary-ng hydrology on the amount, timing, and spatial distribution of

elling 300 (2015) 12–29

fish production in response to more recent changes in hydrology,ALFISH was later modified to a new model, called GEFISH. This illus-trates how model evaluation, with evolving questions and new databecoming available, can lead to the specification of new model(an arrow going back up in the flowchart to Step 16). In particu-lar, GEFISH was aimed at estimating the spatial concentrations offish during falling water levels in the complex ridge-and-sloughregion of the Everglades, which required higher spatial resolu-tion than used in ALFISH (Yurek et al., 2013). Many of the samesteps in developing ALFISH were repeated for GEFISH, though onlya few will be mentioned specifically here. The basic structure ofGEFISH was similar to that of ALFISH, except that lower trophiclevels without size-structure of the fish (illustrates simplificationof the model) are modeled in GEFISH. Finer spatial resolution ofsub-daily water depths was also possible from improved hydrologicmodeling. GEFISH represented three key ecosystem processes in ageneralized freshwater Everglades marsh: (1) biomass productionof fish functional groups, which are linked through predator–preyinteractions with aquatic invertebrates and periphyton, (2) disper-sal of biomass into newly flooded areas in response to the abioticimpetus of changing water depths, and (3) the return movement ofbiomass to deeper areas as waters recede during the dry season,including the possibility of biomass being trapped and concen-trated in local depressions. Unlike ALFISH, GEFISH was not appliedto the whole freshwater Everglades, but to various subareas of theEverglades, with different fish functional types defined by theirgrowth rates and movement strategies.

The data for GEFISH (Step 11) were based on empirical research(e.g., Trexler et al., 2002) and were synthesized into a database. Inparticular, the database included the results of a monitoring pro-tocol that combined estimates of the densities of small fish andmacroinvertebrates with estimates of their encounter rates and netdirectional displacements measured at sampling drift fences to pro-vide information on the movement of these animals at a landscapescale (Obaza et al., 2011). GEFISH simulations were then used tostudy the effects of the seasonal depths and hydroperiod on theability of fish to exploit the flooding landscape to build populationsand quantify prey availability to wading birds (repeating the stepsfrom step 16) but with a new model.

5. Concluding remarks

Effective large-scale restoration is critical for proper use andconservation of the natural resources and ecosystem services pro-vided by these large ecosystems. Yet, the combination of highcosts and high scientific uncertainty associated with the popu-lation dynamics of ecologically, economically, and protected fishspecies and ecosystem health, almost always lead to controversyand debate (see Doyle and Drew, 2008). An integral part of theanalyses usually focuses on the fish species, and on the modelsused to predict population and food web responses to managementactions related to restoration. Based on our experiences in a varietyof ecosystems, we offered a detailed scheme of steps for best (moreaccurately very good) practices for selecting, applying, interpret-ing, and reporting on fish modeling designed to assess the effects ofrestoration. Our scheme is not perfect and should be tweaked andadjusted for specific situations. Also, our scheme could be easilyadapted for most situations involving model predictions or fore-casting of future conditions under other conditions then restorationactions (e.g., climate change effects).

There are several themes that arise in multiple places in our

ENT 8

best practices scheme and associated concepts that are often eitherinadequately documented or not given enough attention. Thesethemes are: clear comparison of how calibration and validationconditions relate to scenario or management conditions (i.e., within

l Mod

tctttptrw

ttglaaFfrm

Ds

MEN

K.A. Rose et al. / Ecologica

he domain of applicability), repeated check-backs to the code andonceptual models, how other models’ outputs are used as inputo the fish models, availability of comprehensive documentation ofhe model, treatment of uncertainty and stochasticity relative tohe real world, and implicit versus explicit representations of keyrocesses. While all of the steps in the scheme are important, thesehemes should be given extra care in their treatment as, in our expe-ience, they are often treated too lightly but attract controversyhen model results are interpreted and reported.

Use of our proposed scheme will require investment of addi-ional time and effort (and dollars) to be done effectively. We arguehat such an investment by the modelers, RRA, stakeholders, andeneral public is well worth it and will more than pay back in theong run in effective and efficient restoration actions and likelyvoided controversies and legal proceedings. Careful, thoughtful,nd well documented modeling is an excellent example of Benranklin’s “an ounce of prevention is worth a pound of cure.” Inef-

ATTACH

ective use of models leads to delays in designing and implementingestoration plans, and less than ecologically and economically opti-al restoration actions ultimately being taken.

etails about the steps in the best practices scheme for the development and applicatiohown in Fig. 1.

Step Description

(1) Know the restoration plan(Concepts 1 and 4)

Understand the details of the various proposeand 4) and general ecosystem health and issuefforts and the historical development that legroups, regulatory and resource agencies (RRWater Act).

(2) Verify how fish modelingresults will be used by theRRA (Concept 5)

Modeling results can be used in a variety of wdecision-making about general trends and ecprecise and accurate predictions of future conforecasts of absolute abundances and biomasrestoration actions) (Concept 5). Also importapredictions is needed. This will affect how easmodeling.

(3) Define the questions to beanswered by the modeling(Concepts 2, 3, and 11)

Defining the questions is a critical step that eninitial generation will be relevant (Concept 3)because of the lack of specification of the quewhat the modeling can do. Explaining the certhe more likely a model can be configured thamany questions is another way the power of

different models to address different subsets

A hypothetical illustration of a poorly and weshrimp, versus (2) How does the wetland-relaconfiguration, inundation frequency) in regioSeptember over the next 20 years.

(4)–(6) Create the conceptualmodels (Concepts 1, 5, 6, 7,and 12)

Conceptual models will provide an important(except the modelers) will not understand thconceptual models help in explaining the momodels require specification of what factors aand effect relationships.Three separate conceptual models should be

(Concept 12), a second model for the factors amodel for how the various restoration actionsspecies of focus (Concepts 1, 5, 7). The overviecontext of the food web and ecosystem. The scommunity of interest, and the third model isaction. It is important to keep these three conback to either being motivated by needing to

or needed to better simulate the restoration aThe conceptual models should define the systlevels of aggregation across time (yearly, monoverview model can be less specific than the

overview conceptual model. The models of thand detailed and will change as the numericaspecies and their life cycles (Concept 1), and trates and the effects of the restoration actionstaken to describe whether representation of tmodel (Concept 6). Also important is definingtime step and number of years, and documenrecruitment, or truly full life cycle).

elling 300 (2015) 12–29 25

Acknowledgements

This paper evolved from a variety of workshops involving someof the authors on how to use ecological and fish models, a report forthe California Delta Science Program on how to develop and imple-ment salmon life cycle models (with James Anderson, MichelleMcClure, and Greg Ruggerone), and a report prepared by KAR andSS and funded by Louisiana’s Coastal Protection and RestorationAuthority on how to select fish models for evaluating restorationoptions for the 2017 Coastal Master Plan. KAR and SS want to thankThe Water Institute of the Gulf for their funding support to writethis paper. Statement of Author Contributions: KAR and SS devel-oped the overall scheme; DLD, SY, and JCT provided the Evergladesexample; WG provided the Colorado River example; DJR workedwith KAR and SS on the Coastal Louisiana example; all authorscontributed to the final document.

T 8

Appendix A.

n of fish models for evaluation of large-scale restoration plans. The steps are also

d restoration actions and the past and current status of key species (Concepts 1es. Often overlooked, it is also important to know the history of the restorationad to the current plan under evaluation. Be familiar with the various stakeholderA), permitting process, and any legal issues (e.g., Endangered Species Act; Clean

ays and at a range of levels of specificity. Results can be limited to informingosystem health, but more likely, the modeling results will be viewed as highlyditions. Clarity is needed in whether the modeling results are to be viewed as

s or as predictions of relative changes (e.g., between without and with thent is clarifying to what degree mechanistic understanding of the modelily and directly empirical relationships (e.g., regression results) are used in the

sures the modeling results that will be reported months to years from their and will meet people’s expectations. Often, modeling is considered unsuccessfulstions to be answered, coupled with people having overly high expectations oftainties and uncertainties is vital (Concept 2). The more specific the questions,t will answer that particular question. Trying to have a single model answer

the modeling gets diluted. Sometimes the appropriate approach is to developof the overarching questions (Concept 11).ll-stated questions would be: (1) What are the effects of wetland creation onted habitat created by projects A, B, and C (i.e., acreages, land-water

n X combine to affect annual shrimp summertime growth and abundance in

communication tool for explaining how the model works. Most everyonee heavy mathematics and the computer codes that are actually the model. Thedels. Simply showing a box and arrow diagram is not sufficient. Conceptualre being considered important (and unimportant by omission) and the cause

specified: an overview model of the major food web dynamics in the systemffecting the population or community dynamics of species of focus, and a third

affect the vital rates of growth, reproduction, mortality, and movement of thew conceptual model allows the other two models to be viewed in the broaderecond model is about what is needed for realistic simulation of the population or

about what is needed for realistic simulation of the effects of the restorationceptual models separate so that later steps of model development can be tracedadd food web complexity, needing to better simulate the population dynamics,ctions.em in terms of the boundaries and the inputs to the model, and describe thethly, daily) and space (domain, resolution) for the system (Concept 5). The

other two models, as there will not be a numerical model that matches thee populations of interest and restoration action effects should be very specificl model is developed. These conceptual models involve specifying the keyhe key factors (e.g., temperature, predation) that affect the fish (life stage) vital

on the vital rates (i.e., the cause-effect relationships) (Concept 5). Care must behese environmental and biological factors will be explicit or implicit in the

the scales for modeling environmental and biological processes including theting how reproduction will be done (single year, multiple years with forced

26 K.A. Rose et al. / Ecological Modelling 300 (2015) 12–29

Step Description

(7) Unify the conceptualmodels

The two conceptual models directly tied to the numerical model should be overlain onto a single conceptual model so the linkbetween restoration actions and the factors affecting the populations and communities are clear. Identification of factors consideredimportant in both, and those needed for only realistic simulation of population dynamics versus restoration effects should be noted.

(8) Review This first review is by the RRA and stakeholders (S) to understand the conceptual models. Review by a scientific group is optionalhere because usually the conceptual models rely heavily on local knowledge. A response document that explains how commentsand suggestions were dealt with is prepared by the modelers with the approval of the RRA.

(9) Identify managementmodels

Management models are part of the model library but they also need special attention. Common examples of management modelsare stock assessment fisheries models and population dynamics models used in Biological Opinions. These models have received agreat deal of scrutiny and will persist, and so any modeling done to evaluate restoration actions should be compared to the existingmanagement models to ensure consistency in inputs, process representations, and outputs that overlap. The management andrestoration models do not need to be the same formulations, as the questions are different, but ensuring some level of consistencywill make later reporting of modeling results easier. People will refer back to the results of the management models and comparethe restoration results to the management model results.

(10) Develop library of modelsand approaches

Develop a summary of existing models and approaches that could be in anyway suitable candidate models to be used, andsummarize them based on their existing versions for comparison across models to help with model selection. Suitability is based onthe capability to modify the model or approach to match the conceptual models, and to answer the questions. We use the term“approaches” here because there are often models developed for other systems or other reasons that could be adapted, and theseshould be included. The library should not be limited to only those models that exist that can be used as is without anymodification. One should err on including models and approaches at this step, even if the models can only address parts ofquestions or were previously used in very different ecosystems. The summary of these models and approaches can be brief. Thislibrary forms the universe of models and approaches from which the actually used models will emerge.

(11) Create data inventory andsummarize prior knowledge

A careful and comprehensive review and synthesis of available data should be done in parallel to developing the model library. Thedata can be monitoring data, results of short-term field studies, laboratory results, and the inputs and outputs of other models. Thedata should be organized according to its potential use to the fish modeling (e.g., inputs, specification of equations, or comparison tomodel outputs), and how it relates to the conceptual models. There is a tendency to be overly broad and vague about how outputsfrom other models will be used as inputs to this modeling (e.g., “output of the vegetation model will provide habitat information tothe fish model”). The temporal and spatial scales of the model outputs to be used as inputs should be carefully described, includingany averaging or other methods to make their spatial and temporal scales match the scales of the receiving fish model.

(12) Candidate models andapproaches

This step reviews the models and approaches in the model library and states the reasons for dismissing the various models andapproaches from further consideration. This is based on whether there are models or approaches available that are obviously notviable and others that are clearly good candidates.

(13) Fewer viable models andapproaches [Exit option]

The model library is combined with the data inventory to determine which of the initially screened models and approaches can besufficiently supported with the available data. It is possible that there are no existing models or approaches that can both answerthe questions and have sufficient credibility with the available data. In that case, the correct decision to exit, with the option todevelop a new model that is not part of the model library.

(14) Constraints Constraints are defined here but not used yet. Constraints here refer to the practical considerations involved with implementing amodel, such as monetary budgets, time schedule of when results are needed, and the availability of expertise and computerresources. Computing considerations include the format and access to source code, data transfer compatibility with the othermodels that will provide inputs to the fish models, and computing speed relative to simulations needed for predictions of scenariosand assessment of uncertainty.

(15) Even fewer viable modelsand approaches [Exit option]

Constraints (except computing speed) are now overlaid on the viable models and approaches. The mixing of selecting models basedon their scientific capabilities versus constraints leads to confusion. Knowing a model or approach was dismissed due to constraints(versus science) is very important so if the constraints change or no models are deemed viable, one can return to the models andapproaches considered scientifically viable before the constraints were factored in. We do not recommend including computerspeed as a constraint here, but rather wait until the next step. The reason is that not enough is known about the model yet toconclude there is insufficient computing power, and modelers (most of which are not formally trained in the latest computingtechnology) often underestimate computing options and power.Also, the correct decision may be to exit or conclude a new model needs to be developed.

(16) Specify the model(s) (allconcepts)

This step is a major effort and could itself be described as a complicated series of iterative steps. The goal here is at least one but notmore than three models or approaches. Major activities include: defining the temporal and spatial scales, the state variables, thefunctional forms for the governing equations, estimating the values of parameters (calibration), and validation. Also important arehow multiple year simulations will be done, the implicit and explicit representations for processes and restoration effects, and howdensity-dependence and variability will be accounted for. Frequent referral and updating to the conceptual models is critical.Decisions about software and hardware are done in parallel with model specification. Access and ability to modify the source codeis critical. To say later that “the model did not let me change the equations to represent the restoration effect the way I wanted to” isunacceptable. It is also unacceptable to make decisions about the model equations and spatial and temporal resolution inanticipation that model simulations will require too much computer time. The model should be specified based on scientificknowledge, addressing the questions, data availability, and constraints. Then the issue of computing can be discussed. It is importantto describe the “ideal” science-based model without adjusting for perceived computing limitations. There are real computinglimitations in some situations, but often these limitations are less restrictive than the ecological and fish modelers consider.All models use some form of mathematics, and the mathematics of the model must be stated. Most all of the numerical fish modelsuse either differential or difference equations.

(17) Prepare a strategydocument

A modeling strategy document would contain a model description, how the model will be calibrated and validated, and how therestoration actions will be simulated. A strategy document is very useful for clearly communicating to people these critical steps ofcalibration, validation, and scenario analysis, and for managing expectations as to what the modeling can solve. The results ofcalibration, validation, and scenario analyses should not be reported in the strategy document.

(18) Review This is the second review, and is usually best done by representatives from the RRA (R), scientific (P), and stakeholders (S). Thestrategy document is a major point at which buy-in is needed from all groups. Only very preliminary results, if any, should bereported in the strategy document that is reviewed, as the focus is on the methods not results. A response document that explainshow comments and suggestions from the review(s) were dealt with is prepared by the modelers with the approval of the RRA.

(19) Perform verification anddiagnostic testing (Concept9)

Verification is the confirmation that the model code is correct (i.e., the equations are represented as described). Verification is oftenoverlooked and reporting that the model generated realistic results is not sufficient to conclude that the model is verified.Verification also includes demonstration that the solution is of sufficient numerical accuracy. Convergence of any numericalsolution must be established (e.g., the time step in a 4th order Runge–Kutta solution). There can be confusion about whether one iscorrectly solving a set of difference equations or poorly solving a set of differential equations. There can also be confusion betweenthe time step for numerical solution and the time step for outputting results.

ATTACHMENT 8

K.A. Rose et al. / Ecological Modelling 300 (2015) 12–29 27

Step Description

(20) Perform calibration(Concept 9)

Calibration is the adjustment of model parameters to get selected output variables to match observed values. Whenever possible,use of automatic methods, rather than ad-hoc adjustments, are preferred. Automatic methods enable the calibration process to berepeated, often include goodness-of-fit measures and also measures of uncertainty, and remove some degree of user judgmentfrom the calibration. There is a tendency to treat the data as truth and without any observation error; both the model outputs anddata should be viewed as approximations to the real system.Calibration should not only focus on the outputs for which there is quantitative data. Evaluating the model qualitatively (e.g.,similar spatial distributions) is also very useful to calibration.Care must be used to document how the output variables used in calibration match (or do not match) the variables that will beused to assess the effects of restoration actions. For example, one can calibrate to fish growth (weight-at-age) under relativelyconstant salinity conditions (food and temperature varied), and then want to use the model to predict how changes in salinity willaffect growth but also spatial distributions. Simply saying the model has been calibrated is not sufficient; the correspondencebetween calibration and restoration outputs must be stated.Calibration often results in changes to model equations (not just parameter values), which means the conceptual models need tobe updated and model verification must be repeated.

(21) Perform validation [Exitoption] (Concept 9)

Validation is the testing of model behavior and prediction outputs with data not used in the set-up of the model or its calibration.Like calibration, the output variables used in the validation must be cross-referenced to the output variables used to assess theeffects of restoration actions, and adjustments may be needed to equations. Inadequate validation results can trigger changes toparameters that require calibration again, and changes to model equations that trigger updating the conceptual models andrepeating verification and calibration. There is an off ramp here to exit if, after multiple attempts, the model cannot be validatedto the extent deemed necessary. Exiting here goes to the post-audit.

(22) Perform sensitivity anduncertainty analysis(Concepts 2 and 10)

Sensitivity and uncertainty analyses are often confused, and people are unduly attracted to the ideas. All models will be sensitiveto some inputs relative to others, and all will generate predictions with uncertainty (Concept 10). It is easy to use the idea ofsensitivity and uncertainty analysis to somehow foster additional confidence in the model predictions when, in fact, the analysesare often misinterpreted and do not result in any changes to the model. When used appropriately and the sources of variabilitybeing used to vary the parameters are carefully considered and documented (Concept 2), sensitivity and uncertainty analyses areextremely useful for refining the model and correctly interpreting model predictions.

(23) Report on results forbaseline only

This interim report documents the results of verification, calibration, and validation but not the results of scenario analyses.Judging the model credibility should focus on baseline conditions because if provided with scenario results, people tend to judgethe model as to whether they agree with the model-predicted effects of the restoration actions. The idea is to confirm theconceptual model and baseline model configuration about factors important in population and community dynamics. The laterreviews will include the results of scenarios.

(24) Review This is the third review and involves RRA and science reviews (P); the role of stakeholders in judging the baseline results is oftentoo focused on specific issues rather than the overall model performance.

(25) Scenarios There will always some type of baseline simulation against which the effects of restoration actions will be compared.Comparisons between simulations without and with the restoration actions should be done so that the only difference betweenthem is the restoration actions. Typically, the baseline is some form of “future without action.” However, what that means needsto be defined and a rationale provided. Often, simulations are for multiple years and decades, and issues like how continuedhuman growth and development and climate change are ignored or incorporated into the future conditions becomes important.

(26) Uncertainty analysis Uncertainty analysis is repeated here because it is likely a different analysis to compare the future without and with restorationactions than the earlier uncertainty analysis. The focus here on the scenarios, and could include uncertainties aboutimplementation.

(27) Results delivered to RRA Delivery and communication of the results to the RRA for their use and likely inclusion in reports should be treated as a majorstep. It is easy for the modeling results to be inadvertently mis-construed because of poor transfer of information betweenmodelers and the receiving RRA, especially when the RRA personnel are not formally trained in ecological and fish modeling. Thisstep occurs near at the end and thus the communication of modeling results can be rushed due to tight reporting schedules andmoney running out.

(28) Review This is the third review but also uses almost the same information as the fourth review, but to a different audience. This thirdreview is the delivery of the results to the RRA, who can then do an internal evaluation of the results. This gives the RRA anopportunity to examine the results before stakeholders.

(29) Public reporting Once the RRA has had an opportunity to comment on the modeling results a report is usually produced that includes the modelingresults but is also broader in scope and likely includes information (e.g., costs, decisions).

(30) Review This is the fourth review and uses almost the same information as the third review but uses stakeholders (S) to evaluate the publicreport.

(31) Post-auditing Post-auditing is documenting the model in its final form for the analyses and how it developed throughout the analysis, and alsorevisiting some of the major steps to ensure all information is up to date. Post-auditing is a combination of archiving and

ailableme reears, t

R

A

A

A

A

A

B

B

ATTACHMENT 8

back-checking to see if any new data are avPost-auditing is extremely valuable if the sapurposes. Some regulatory processes take y

eferences

ble, K.W., 2005. A re-examination of fish estuarine dependence: evidence for con-nectivity between estuarine and ocean habitats. Estuarine Coast. Shelf Sci. 64,5–17.

hmadi-Nedushan, B., St-Hilaire, A., Bérubé, M., Robichaud, É., Thiémonge, N., Bobée,B., 2006. A review of statistical methods for the evaluation of aquatic habitatsuitability for instream flow assessment. River Res. Appl. 22, 503–523.

tkins, 2011. Scientific Assessment of Two Dam Removal Alternatives on CohoSalmon and Steelhead. Atkins, Portland, OR.

tkins, 2012. Peer Review Panel Report on Draft Klamath Dam Removal OverviewReport for the Secretary of the Interior. Atkins, Portland, OR (Prepared for theUnited States Department of the Interior).

ult, J.S., Luo, J., Smith, S.G., Serafy, J.E., Wang, J.D., Humston, R., Diaz, G.A., 1999. Aspatial dynamic multistock production model. Can. J. Fish. Aquat. Sci. 56 (Suppl.

1), 4–25.

alon, E.K., 1979. The theory of saltation and its application in the ontogeny of fishes:steps and thresholds. Environ. Biol. Fishes 4, 97–101.

arbier, E.B., Hacker, S.D., Kennedy, C., Koch, E.W., Stier, A.C., Silliman, B.R., 2011. Thevalue of estuarine and coastal ecosystem services. Ecol. Monogr. 81, 169–193.

or if the conceptual models have changed based on new information.storation issues arise in the future or if others want to use the model for theirhen there is a hiatus period, and then the issues arise again.

Barlas, Y., Carpenter, S., 1990. Philosophical roots of model validation: twoparadigms. Syst. Dyn. Rev. 6, 148–166.

Barnthouse, L.W., Boreman, J., Christensen, S.W., Goodyear, C.P., Van Winkle, W.,Vaughan, D.S., 1984. Population biology in the courtroom: the Hudson Rivercontroversy. Bioscience 34, 14–19.

Barnthouse, L.W., 2004. Extrapolating impingement and entrainment losses toequivalent adults and production foregone. In: EPRI Report 1008471. ElectricPower Research Institute, Palo Alto, CA.

Barnthouse, L.W., 2005. Parameter development for equivalent adult and productionforegone models. In: EPRI Report 1008832. Electric Power Research Institute,Palo Alto, CA.

Bartell, S.M., Nair, S.K., Wu, Y., 2010. Mississippi River-Gulf Outlet (MRGO) EcosystemRestoration Study: Preliminary Evaluation of Proposed Freshwater Diversionin the Vicinity of Violet, Louisiana. E2 Consulting Engineers, New Orleans,LA (Prepared for the United States Army Corps of Engineer-New Orleans

District).

Baxter, R., Brown, L.R., Castillo, G., Conrad, L., Culberson, S., Dekar, M., Feyrer,F., Gehrts, K., Grimaldo, L., Hunt, T., Kirsch, J., Mueller-Solger, A., Nobriga,M., Slater, S., Sommer, T., Souza, K., 2015. An Updated Conceptual Model forDelta Smelt: Our Evolving Understanding of an Estuarine Fish. Interagency

2 l Mod

B

B

B

B

C

C

C

C

C

C

C

C

D

D

D

D

D

d

d

D

D

D

D

E

E

E

E

F

F

CHM

8 K.A. Rose et al. / Ecologica

Ecological Program Technical Report 90. California Department of WaterResources, Sacramento, CA.

eck, M.W., Heck, K.L., Able, K.W., Childers, D.L., Eggleston, D.B., Gillanders, B.M.,Halpern, B., Hays, C.G., Hoshino, K., Minello, T.J., Orth, R.J., Sheridan, P.F., Wein-stein, M.P., 2001. The identification, conservation, and management of estuarineand marine nurseries for fish and invertebrates. Bioscience 51, 633–641.

jørkvoll, E., Grøtan, V., Aanes, S., Sæther, B.E., Engen, S., Aanes, R., 2012. Stochasticpopulation dynamics and life-history variation in marine fish species. Am. Nat.180, 372–387.

jørnstad, O.N., Grenfell, B.T., 2001. Noisy clockwork: time series analysis of popu-lation fluctuations in animals. Science 293, 638–643.

ullock, J.M., Aronson, J., Newton, A.C., Pywell, R.F., Rey-Benayas, J.M., 2011. Restora-tion of ecosystem services and biodiversity: conflicts and opportunities. TrendsEcol. Evol. 26, 541–549.

ADWR (California Department of Water Resources), 2013. Bay Delta Conserva-tion Plan. Public Draft. November. Prepared by ICF International (ICF 00343.12),Sacramento, CA.

affey, R.H., Schexnayder, M., 2002. Fisheries implications of freshwater reintroduc-tions. In: Coastal Wetland Planning Protection and Restoration Act InterpretiveTopic Series on Coastal Wetland Restoration in Louisiana, National Sea GrantLibrary No. LSU-G-02-003, 8 pp.

aswell, H., 2001. Matrix Population Models: Construction, Analysis, and Interpre-tation, second ed. Sinauer Associates, Sunderland, MA.

hiu, G.S., 2013. Food web modeling. In: El-Shaarawi, A.H., Piegorsch, W.W. (Eds.),Encyclopedia of Environmetrics. Wiley-Blackwell, Hoboken, NJ, pp. 1063–1066.

lark, J.S., Carpenter, S., Barber, M., Collins, S., Dobson, A., Foley, J., Lodge, D., Pascual,M., Pielke, R., Pizer, W., Pringle, C., Reid, W., Rose, K., Sala, O., Schlesinger, W.,Wall, D., Wear, D., 2001. Ecological forecasts: an emerging imperative. Science293, 657–660.

owan, J.H., Shaw, R.F., 2002. Recruitment. In: Fuiman, L.A., Werner, R.G. (Eds.), Fish-ery Science: The Unique Contributions of Early Life Stages. Blackwell Science,Oxford, UK, pp. 88–111.

owen, R.K., Sponaugle, S., 2009. Larval dispersal and marine population connectiv-ity. Annu. Rev. Mar. Sci. 1, 443–466.

ushing, D.H., 1996. Towards a science of recruitment in fish populations. Excellencein Ecology, vol. 7. Ecology Institute, Oldendorf/Luhe, Germany.

eAngelis, D.L., 2012. The case for ratio dependence in trophic ecology. Trends Ecol.Evol. 28, 259–260.

eAngelis, D.L., Gross, L.J., Huston, M.A., Wolff, W.F., Fleming, D.M., Comiskey, E.J.,Sylvester, S., 1998. Landscape modeling for Everglades ecosystem restoration.Ecosystems 1, 64–75.

eAngelis, D.L., Loftus, W.F., Trexler, J.C., Ulanowicz, R.E., 1997. Modeling fishdynamics and effects of stress in a hydrologically pulsed ecosystem. J. Aquat.Ecosyst. Stress Recovery 1, 1–13.

eAngelis, D.L., Mooij, W.M., 2005. Individual-based modeling of ecological andevolutionary processes. Annu. Rev. Ecol. Evol. Syst. 36, 147–168.

eAngelis, D.L., Rose, K.A., 1992. Which individual-based approach is most appro-priate for a given problem? In: DeAngelis, D.L., Gross, L.J. (Eds.), Individual-BasedModels and Approaches in Ecology. Chapman and Hall, New York, pp. 67–87.

e Mutsert, K., Cowan, J.H., Walters, C.J., 2012. Using ecopath with ecosim to explorenekton community response to freshwater diversion into a Louisiana estuary.Mar. Coast. Fish.: Dyn. Manage. Ecosyst. Sci. 4, 104–116.

eYoung, B., Barange, M., Beaugrand, G., Harris, R., Perry, R.I., Scheffer, M., Werner,F., 2008. Regime shifts in marine ecosystems: detection, prediction and man-agement. Trends Ecol. Evol. 23, 402–409.

ickey, T.D., 2003. Emerging ocean observations for interdisciplinary data assimila-tion systems. J. Mar. Syst. 40, 5–48.

iGennaro, B., Reed, D., Swanson, C., Hastings, L., Hymanson, Z., Healey, M., Siegel,S., Cantrell, S., Herbold, B., 2012. Using conceptual models and decision-supporttools to guide ecosystem restoration planning and adaptive management: anexample from the Sacramento–San Joaquin Delta, California. San Franc. EstuaryWatershed Sci. 10 (3).

oyle, M., Drew, C.A., 2008. Large-Scale Ecosystem Restoration. Island Press, Wash-ington, DC.

raugelis-Dale, R., 2008. Assessment of effectiveness and limitations of habitat suit-ability models for wetland restoration. In: U.S. Geological Survey Open-FileReport 2007-1254., pp. 136 p.

lith, J., Burgman, M.A., 2003. Habitat models for population viability analysis.In: Brigham, C.A., Schwartz, M.W. (Eds.), Population Viability in Plants: Con-servation, Management, and Modeling of Rare Plants. Ecological Studies 165.Springer-Verlag, Netherlands, pp. 145–167.

ngle, V.D., 2011. Estimating the provision of ecosystem services by Gulf of Mexicocoastal wetlands. Wetlands 31, 179–193.

spinoza-Tenorio, A., Wolff, M., Taylor, M.H., Espejel, I., 2012. What model suitsecosystem-based fisheries management? A plea for a structured modeling pro-cess. Rev. Fish Biol. Fish. 22, 81–94.

vans, M.R., Grimm, V., Johst, K., Knuuttila, T., de Langhe, R., Lessells, C.M., Merz,M., O’Malley, M.A., Orzack, S.H., Weisberg, M., Wilkinson, D.J., Wolkenhauer, O.,Benton, T.G., 2013. Do simple models lead to generality in ecology? Trends Ecol.Evol. 28, 578–583.

AO, 2008. Fisheries management. 2. The ecosystem approach to fisheries. 2.1 Best

ATTA

practices in ecosystem modelling for informing an ecosystem approach to fish-eries. In: FAO Fisheries Technical Guidelines for Responsible Fisheries. Food andAgriculture Organization, Rome, pp. 78 p (No. 4, Suppl. 2, Add. 1).

ath, B.D., Jorgensen, S.E., Scharler, U.M., 2011. Ecological modeling in environmen-tal management: history and applications. In: Wolanski, E., McLusky, D. (Eds.),

elling 300 (2015) 12–29

Treatise on Estuarine and Coastal Science. Academic Press, Waltham, MA, pp.23–33.

Fennema, R.J., Neidrauer, C.J., Johnson, R.A., MacVicar, T.K., Perkins, W.A., 1994.A computer model to simulate natural Everglades hydrology. In: Davis, S.M.,Ogden, J.C. (Eds.), Everglades: The Ecosystem and its Restoration. St. Lucie Press,Delray Beach, FL, pp. 249–289.

Ferson, S., Ginzburg, L.R., 1996. Different methods are needed to propagate ignoranceand variability. Reliab. Eng. Syst. Saf. 54, 133–144.

Fischenich, J.C., 2008. The Application of Conceptual Models to Ecosystem Restora-tion. In: Engineer Research and Development Center Report ERDC/EBA TN-08-1.United States Army Corps of Engineers.

Fuller, M.M., Gross, L.J., Duke-Sylvester, S.M., Palmer, M., 2008. Testing the robust-ness of management decisions to uncertainty: everglades restoration scenarios.Ecol. Appl. 18, 711–723.

Gaff, H.D., Chick, J., Salinas, R., Gross, L.J., 1999. Evaluation and Modifications ofATLSS. In: Unpublished Report, Across Trophic Level System Simulation (ATLSS)Project. University of Tennessee, Knoxville, TN.

Gaff, H., Chick, J., Trexler, J., DeAngelis, D., Gross, L., Salinas, R., 2004. Evaluation ofand insights from ALFISH: a spatially explicit landscape-level simulation of fishpopulations in the Everglades. Hydrobiologia 520, 73–87.

Gaff, H., DeAngelis, D.L., Gross, L.J., Salinas, R., Shorrosh, M., 2000. A dynamics land-scape model for fish in the Everglades and its application to restoration. Ecol.Modell. 127, 33–52.

Gårdmark, A., Lindegren, M., Neuenfeldt, S., Blenckner, T., Heikinheimo, O., Müller-Karulis, B., Niiranen, S., Tomczak, M.T., Aro, E., Wikstrom, A., Möllmann, C., 2013.Biological ensemble modeling to evaluate potential futures of living marineresources. Ecol. Appl. 23, 742–754.

Gardner, R.H., Jager, H.I., Petersen, J.H., Rose, K.A., 2002. Review and Evaluation of:Greater Everglades Ecosystem Restoration Ecological Processes and ModelingWorkshop, May 9–10, 2002, Fort Lauderdale, FL.

Gentile, J.H., Harwell, M.A., Cropper, W., Harwell, C.C., DeAngelis, D., Davis, S., Ogden,J.C., Lirman, D., 2001. Ecological conceptual models: a framework and case studyon ecosystem management for South Florida sustainability. Sci. Tot. Environ.274, 231–253.

Gore, J.A., Nestler, J.M., 1988. Instream flow studies in perspective. Regul. Rivers:Res. Manage. 2, 93–101.

Graf, W.L., Wohl, E., Sinha, T., Sabo, J.L., 2010. Sedimentation and sustain-ability of western American reservoirs. Water Resour. Res. 46, W12535,http://dx.doi.org/10.1029/2009WR008836.

Gray, A.D., 2013. Chesapeake Bay Fiscal 2014 Budget Overview. Report pre-pared by the Maryland Department of Legislative Services, Annapolis, MD,pp. 36 p.

Grimm, V., Schmidt, E., Wissel, C., 1992. On the application of stability concepts inecology. Ecol. Modell. 63, 143–161.

Grimm, V., Wissel, C., 1997. Babel, or the ecological stability discussions: an inven-tory and analysis of terminology and a guide for avoiding confusion. Oecologia109, 323–334.

Grimm, V., Revilla, E., Berger, U., Jeltsch, F., Mooij, W.M., Railsback, S.F., Thulke, H.-H., Weiner, J., Wiegand, T., DeAngelis, D.L., 2005. Pattern-oriented modeling ofagent-based complex systems: lessons from ecology. Science 310, 987–991.

Harwood, J., Stokes, K., 2003. Coping with uncertainty in ecological advice: lessonsfrom fisheries. Trends Ecol. Evol. 18, 617–622.

Hilborn, R., 2011. Future directions in ecosystem based fisheries management: apersonal perspective. Fish. Res. 108, 235–239.

Hixon, M.A., Pacala, S.W., Sandin, S.A., 2002. Population regulation: historical con-text and contemporary challenges of open vs. closed systems. Ecology 83,1490–1508.

Houde, E.D., 2008. Emerging from Hjort’s shadow. J. Northwest Atl. Fish. Sci. 41,53–70.

Hughes, T.P., 1990. Recruitment limitation, mortality, and population regulation inopen systems: a case study. Ecology 71, 12–20.

Jackson, J.B., Kirby, M.X., Berger, W.H., Bjorndal, K.A., Botsford, L.W., Bourque, B.J.,Bradbury, R.H., Cooke, R., Erlandson, J., Estes, J.A., Hughes, T.P., Kidwell, S.,Lange, C.B., Lenihan, H.S., Pandolfi, J.M., Peterson, C.H., Steneck, R.S., Tegner,M.J., Warner, R.R., 2001. Historical overfishing and the recent collapse of coastalecosystems. Science 293, 629–637.

Jakeman, A.J., Letcher, R.A., Norton, J.P., 2006. Ten iterative steps in development andevaluation of environmental models. Environ. Modell. Softw. 21, 602–614.

Janssen, P.H.M., Heuberger, P.S.C., 1995. Calibration of process-oriented models.Ecol. Modell. 83, 55–66.

Jones, C.M., 2006. Estuarine and diadromous fish metapopulations. In: Kritzer, J.P.,Sale, P.F. (Eds.), Marine Metapopulations. Elsevier Academic Press, Burlington,MA, pp. 119–154.

Jørgensen, C., Opdal, A.F., Fiksen, Ø., 2013. Can behavioural ecology unite hypothesesfor fish recruitment? ICES J. Mar. Sci. 71, 909–917.

Kaplan, I.C., Horne, P.J., Levin, P.S., 2012. Screening California Current fishery man-agement scenarios using the Atlantis end-to-end ecosystem model. Progr.Oceanogr. 102, 5–18.

Larsen, L.G., Harvey, J.W., Crimaldi, J.P., 2007. A delicate balance: ecohydrologicalfeedbacks governing landscape morphology in a lotic peatland. Ecol. Monogr.77, 591–614.

ENT 8

Letcher, B.H., Rice, J.A., 1997. Prey patchiness and larval fish growth and survival:inferences from an individual-based model. Ecol. Modell. 95, 29–43.

Lett, C., Rose, K.A., Megrey, B.A., 2009. Biophysical models of small pelagic fish. In:Checkley, D., Alheit, J., Oozeki, Y., Roy, C. (Eds.), Climate Change and Small PelagicFish. Cambridge University Press, Cambridge, MA, pp. 88–111.

l Mod

L

L

L

M

M

M

M

M

N

N

N

N

N

N

N

N

O

O

O

P

P

P

P

P

R

R

R

R

R

R

MEN

K.A. Rose et al. / Ecologica

evins, R., 1966. The strategy of model building in population biology. Am. Sci. 54,421–431.

iermann, M., Hilborn, R., 1997. Depensation in fish stocks: a hierarchic Bayesianmeta-analysis. Can. J. Fish. Aquat. Sci. 54, 1976–1984.

ink, J.S., Ihde, T.F., Harvey, C.J., Gaichas, S.K., Field, J.C., Brodziak, J.K.T., Townsend,H.M., Peterman, R.M., 2012. Dealing with uncertainty in ecosystem models: theparadox of use for living marine resource management. Progr. Oceanogr. 102,102–114.

agnusson, A., Punt, A.E., Hilborn, R., 2013. Measuring uncertainty in fisheries stockassessment: the delta method, bootstrap, and MCMC. Fish Fish. 14, 325–342.

cCann, K., Hastings, A., Huxel, G.R., 1998. Weak trophic interactions and the balanceof nature. Nature 395, 794–798.

cCann, K., Shuter, B., 1997. Bioenergetics of life history strategies and the compar-ative allometry of reproduction. Can. J. Fish. Aquat. Sci. 54, 1289–1298.

cKinley, S., Pine, B., Weitkamp, D., 2011. Science Review of Testimony in the DeltaSmelt Cases: Summary Report. RESOLVE (Atkins), Washington, DC.

inckley, W.L., 1991. Native fishes of the Grand Canyon region: an obituary? In:Colorado River Ecology and Dam Management. National Academy Press, Wash-ington, DC, pp. 124–177.

oble, R.A.A., Bredeweg, B., Linnebank, F., Salles, P., Cowx, I.G., 2009. A qualitativemodel of limiting factors for a salmon life cycle in the context of river rehabili-tation. Ecol. Inform. 4, 299–319.

RC (National Research Council), 2012a. Progress toward Restoring the Everglades:The Fourth Biennial Review. National Academies Press, Washington, DC.

RC (National Research Council), 2012b. Sustainable Water and EnvironmentalManagement in the California Bay-Delta. National Academies Press, Washing-ton, DC.

RC (National Research Council), 2011. Achieving Nutrient and Sediment Reduc-tion Goals in the Chesapeake Bay: An Evaluation of Program Strategies andImplementation. National Academies Press, Washington, DC.

RC (National Research Council), 2010. A Scientific Assessment of Alternatives forReducing Water Management Effects on Threatened and Endangered Fishes inCalifornia’s Bay Delta. National Academies Press, Washington, DC.

RC (National Research Council), 1996. River Resource Management in the GrandCanyon. National Academies Press, Washington, DC.

RC (National Research Council), 1987. River and Dam Management: A Review of theBureau of Reclamation’s Glen Canyon Environmental Studies. National AcademyPress, Washington, DC.

yman, J.A., Baltz, D.M., Kaller, M.D., Leberg, P.L., Richards, C.P., Romaire, R.P., Soniat,T.M., 2013. Likely changes in habitat quality for fish and wildlife in coastalLouisiana during the next fifty years. J. Coast. Res. 67, 60–74.

baza, A., DeAngelis, D.L., Trexler, J.C., 2011. Using data from an encounter samplerto model fish dispersal. J. Fish Biol. 78, 495–513.

gden, J.C., Davis, S.M., Jacobs, K.J., Barnes, T., Fling, H.E., 2005. The use of conceptualecological models to guide ecosystem restoration in South Florida. Wetlands 25,795–809.

rzack, S.H., 2005. Discussion: what, if anything, is the strategy of model buildingin population biology? A comment on Levins (1966) and Odenbaugh (2003).Philos. Sci. 72, 479–485.

eyronnin, N., Green, M., Richards, C.P., Owens, A., Reed, D., Chamberlain, J., Groves,D.G., Rhinehart, W.K., Belhadjali, K., 2013. Louisiana’s 2012 coastal master plan:overview of a science-based and publicly informed decision-making process. J.Coast. Res. 67, 1–15.

ianka, E.R., 1970. r and K selection or b and d selection? Am. Nat. 106,581–588.

ihl, L., Cattrijsse, A., Codlig, I., Mathieson, S., McLusky, D.S., Roberts, C., 2002.Habitat use by fishes in estuaries and other brackish areas. In: Elliott, M.,Hemingway, K. (Eds.), Fishes in Estuaries. Blackwell Science, Oxford, UK,pp. 10–54.

innegar, J.K., Trenkel, V.M., Blanchard, J.L., 2008. 80 years of multispecies fisheriesmodelling: significant advances and continuing challenges. In: Payne, A., Cotter,J., Potter, T. (Eds.), Advances in Fisheries Science: 50 years on from Beverton andHolt. Blackwell Publishing, Oxford, UK, pp. 325–357.

itchford, J.W., Brindley, J., 2001. Prey patchiness, predator survival and fish recruit-ment. Bull. Math. Biol. 63, 527–546.

efsgaard, J.C., Henriksen, H.J., 2004. Modelling guidelines––terminology and guid-ing principles. Adv. Water Resour. 27, 71–82.

oloff, G.J., Kernohan, B.J., 1999. Evaluating reliability of habitat suitability indexmodels. Wildl. Soc. Bull. 27, 973–985.

ose, K.A., 2000. Why are quantitative relationships between environmental qualityand fish populations so elusive? Ecol. Appl. 10, 367–385.

ose, K.A., Adamack, A.T., Murphy, C.A., Sable, S.E., Kolesar, S.E., Craig, J.K., Breitburg,D.L., Thomas, P., Brouwer, M.H., Cerco, C.F., Diamond, S., 2009. Does hypoxia havepopulation-level effects on coastal fish? Musings from the virtual world. J. Exp.Mar. Biol. Ecol. 381, S188–S203.

ose, K.A., Cowan, J.H., Winemiller, K.O., Myers, R.A., Hilborn, R., 2001. Compensatory

ATTACH

density-dependence in fish populations: importance, controversy, understand-ing, and prognosis. Fish Fish. 2, 293–327.

ose, K.A., Megrey, B.A., Werner, F.E., Ware, D.M., 2007. Calibration of the NEMUROnutrient–phytoplankton–zooplankton food web model to a coastal ecosystem:evaluation of an automated calibration approach. Ecol. Modell. 202, 38–51.

elling 300 (2015) 12–29 29

Rose, K.A., Sable, S.E., 2009. Multispecies modeling of fish populations. In: Megrey,B.A., Moksness, E. (Eds.), Computers in Fisheries Research. , 2nd ed. Springer, pp.373–397.

Rose, K.A., Sable, S.E., 2013. The 2017 Coastal Master Plan: Strategy for SelectingFish Modeling Approaches. Coastal Protection and Restoration Authority, BatonRouge, LA, pp. 122 p.

Roth, B.M., Rose, K.A., Rozas, L.P., Minello, T.J., 2008. Relative influence of habitat frag-mentation and inundation on brown shrimp Farfantepanaeus aztecus productionin northern Gulf of Mexico salt marshes. Mar. Ecol. Progr. Ser. 359, 185–202.

Rykiel, E.J., 1996. Testing ecological models: the meaning of validation. Ecol. Modell.90, 229–244.

Sable, S.E., Rose, K.A., 2010. Predicting yellow perch population responses using adensity-dependent age-structured matrix projection model: how many annualdata points are needed? Trans. Am. Fish. Soc. 139, 1857–1871.

Saltelli, A., Annoni, P., 2010. How to avoid a perfunctory sensitivity analysis. Environ.Modell. Softw. 25, 1508–1517.

Saltelli, A., Tarantola, S., Campolongo, F., Ratto, M., 2004. Sensitivity Analysis in Prac-tice: A Guide to Assessing Scientific Models. John Wiley and Sons, West Sussex,England.

Schmolke, A., Thorbek, P., DeAngelis, D.L., Grimm, V., 2010. Ecological models sup-porting environmental decision making: a strategy for the future. Trends Ecol.Evol. 25, 479–486.

Scheffer, M., Carpenter, S.R., 2003. Catastrophic regime shifts in ecosystems: linkingtheory to observation. Trends Ecol. Evol. 18, 648–656.

Science Sub Group, 1996. South Florida ecosystem restoration: scientific informationneeds. In: Report to the Working Group of the South Florida Ecosystem Restora-tion Task Force. Working Group of the South Florida Ecosystem Restoration TaskForce.

Stow, C.A., Jolliff, J., McGillicuddy, D.J., Doney, S.C., Allen, J., Friedrichs, M.A., Rose, K.A.,Wallhead, P., 2009. Skill assessment for coupled biological/physical models ofmarine systems. J. Mar. Syst. 76, 4–15.

Suding, K.N., 2011. Toward an era of restoration in ecology: successes, failures, andopportunities ahead. Annu. Rev. Ecol. Evol. Syst. 42, 465–487.

Summitt, A.R., 2013. Contested Waters: An Environmental History of the ColoradoRiver. University Press of Colorado, Boulder, CO.

Swartzman, G., 1996. Resource modeling moves into the courtroom. Ecol. Modell.92, 277–288.

Swannack, T.M., Fischenich, J.C., Tazik, D.J., 2012. Ecological modeling guide forecosystem restoration and management. In: Engineer Research and Develop-ment Center Report ERDC/EL TR-12-18. United States Army Corps of Engineers.

Sylvester, S., 1998. Landscape classes and pseudotopography. In: Unpublished ATLSSReport. Across Trophic Level System Simulation (ATLSS) Project. University ofTennessee, Knoxville, TN.

Trexler, J., Loftus, W., Jordan, F., Chick, J., Kandl, K., McElroy, T., Bass, O.L., 2002. Eco-logical scale and its implications for freshwater fishes in the Florida Everglades.In: Porter, J.W., Porter, K.G. (Eds.), The Everglades, Florida Bay, and Coral Reefsof the Florida Keys: An Ecosystem Sourcebook. CRC Press, Boca Raton, FL, pp.153–181.

Turchin, P., 2003. Complex Population Dynamics: A Theoretical/Empirical Synthesis,vol. 35. Princeton University Press, Princeton, NJ.

US DOI, 2012. Klamath Dam Removal Overview Report for the Secretary of the Inte-rior: An Assessment of Science and Technical Information. Report prepared bythe US Department of Interior and the National Marine Fisheries Service of theUS Department of Commerce.

USGS (United States Geological Survey), 2011. Grand Canyon Monitory and ResearchCenter, Online at 〈http://www.gcmrc.gov/about/about default.aspx〉 (accessedJune 22, 2014).

Vila-Gispert, A., Moreno-Amich, R., Garcia-Berthou, E., 2002. Gradients of life-historyvariation: an intercontinental comparison of fishes. Rev. Fish Biol. Fish. 12,417–427.

Wainwright, J., Mulligan, M., 2004. Environmental Modelling: Finding Simplicity inComplexity. John Wiley and Sons, Oxford, UK.

Walters, C., Kitchell, J.F., 2001. Cultivation/depensation effects on juvenile survivaland recruitment: implications for the theory of fishing. Can. J. Fish. Aquat. Sci.58, 39–50.

West, J., Adriance, J., Lewis, K., Powers, J.E., 2013. Assessment of striped mullet inLouisiana waters. In: 2014 Report of the Louisiana Department of Wildlife andFisheries, 31 pp.

Whipple, S.J., Link, J.S., Garrison, L.P., Fogarty, M.J., 2000. Models of predation andfishing mortality in aquatic ecosystems. Fish Fish. 1, 22–40.

Winemiller, K.O., Rose, K.A., 1992. Patterns of life-history diversification in NorthAmerican fishes: implications for population regulation. Can. J. Fish. Aquat. Sci.49, 2196–2218.

Winemiller, K.O., Rose, K.A., 1993. Why do most fish produce many tiny offspring?Am. Nat. 142, 585–603.

Wu, Y., Liu, S., 2012. Automating calibration, sensitivity and uncertainty analysis

T 8

of complex models using the R package Flexible Modeling Environment (FME):SWAT as an example. Environ. Modell. Softw. 31, 99–109.

Yurek, S., DeAngelis, D.L., Trexler, J.C., Jopp, F., Donalson, D.D., 2013. Simulatingmechanisms for dispersal, production and stranding of small forage fish in tem-porary wetland habitats. Ecol. Modell. 250, 391–401.