JAMS - a Software Framework for Integrated Environmental Modelling

Sven Kralisch, Peter Krause and Christian FischerDepartment of Geoinformatics, Hydrology and Modelling, School of Chemical and Earth Sciences,

Friedrich Schiller University, Jena, Germany Email: sven.kralisch@uni-jena.de

AbstractThe pressures on environmental systems are increasing worldwide due to population growth, socio-economic development and the consequences of climate change. Therefore, adaptable tools and methods are needed to elaborate information, develop understanding and create strategies for the sustainable use and management of such systems. These tools should assist scientists, natural resource managers and decision makers by providing information about the system’s state and help them to identify suitable management strategies, estimate their performance and quantify the impact of future development scenarios.

These objectives are tackled by simulation models that can cope with components and processes of environmental systems, e.g. the hydrological and nutrient cycles. Modelling frameworks and accompanying standards that allow for an easy implementation and extension of such models have therefore gained increasing attention from both model developers and users during the last years.

The Jena Adaptable Modelling System (JAMS) is an open-source software framework that has been especially designed to address the demands of process-based environmental model development and various aspects of model application. The model design and implementation stage is supported by (i) the explicit representation of the spatial and temporal domains via appropriate data types, (ii) a strong separation of process simulation components from the description of more complex models, and (iii) the use of annotations to provide detailed meta-information on the purpose of simulation components and the meaning of their input and output data. In addition, JAMS supports the model application stage by providing (i) flexible interfaces for the input and output of environmental data, (ii) powerful parameter optimization and uncertainty analysis methods, (iii) service-oriented runtime environments for remote and grid-based model execution, and (iv) user-friendly functions for the analysis and visualization of environmental data.

Graphical user interfaces support JAMS users during all stages of model development and application. A model building and parameterization front-end assists users during the creation of problem-tailored models and with the definition of model-related parameterization interfaces. The calibration of existing models is supported by a special graphical assistant which guides users during the setup and application of different state-of-the-art parameter optimization algorithms. A data explorer finally facilitates the analysis of JAMS models by providing plot and map visualizations of a model's input/result data and with the MCAT5 toolbox that offers powerful functionalities for the sensitivity analysis of environmental simulation models. In addition, a web-based user interface allows the definition and deployment of server-side model simulation and calibration tasks, thus providing easy access to high performance computing facilities.

This paper will present the JAMS core concepts and give an introduction to models, simulation components and supporting tools already available for that framework. Sample applications will be

used to underline the advantages of component-based model designs and to show how JAMS can be used to address the challenges of integrated environmental modelling.

1 IntroductionTo tackle problems of integrated water resources management, like e.g. raised by the implementation of the European Water Framework Directive, flexible and powerful simulation models are needed in order to analyse the current state of environmental systems or to project their future dynamics under predefined development scenarios. Especially in cases where domain-specific models cannot easily be extended to meet given modelling requirements, model coupling is a promising approach to create integrated solutions for the detailed consideration of a wide variety of different environmental processes. The systems that have emerged during the last two decades to address this requirement range from pure interface solutions that focus more on coupling of already existing models to simulation frameworks that also cover the creation of problem-tailored simulation components along given requirements. Systems of the first group are usually less focused on specific scientific domains and applications and rather provide standardized interfaces that existing models can be adapted to, allowing for broader and more standardized model coupling approaches. Examples are the Common Component Architecture (CCA), the High Level Architecture (HLA) or the Open Modeling Interface (OpenMI). Systems of the second group usually provide a software architecture (e.g. by means of interfaces and abstract classes) that (i) specifies how modelling components inside the framework need to be structured and how they can interact with each other, and (ii) that controls the creation, linkage and execution of components. While some of them also support the integration of existing, legacy components (e.g. by using code wrapping) these systems usually focus on the implementation of framework-compliant code. Examples are the Modular Modelling System (MMS), the Object Modelling System (OMS), The Invisible Modelling Environment (TIME) or LIQUID, all of them being specialized in the creation of hydrology-related models. An overview of state-of-the-art modelling frameworks can be found in Jagers (2010).

The objective of this paper is to present the Jena Adaptable Modelling System (JAMS). Originally developed with a focus on hydrological process simulation, it is a framework for environmental simulation model development and application in general (Kralisch and Krause 2006). JAMS aims on simulating environmental processes at discrete points in time and/or space. Such systems, often referred to as timed event system, are widely-used by many distributed hydrological models applied in current practice. JAMS is developed as open source software under the GNU Lesser General Public License and maintained at the Department for Geoinformatics, Hydrology and Modelling of the Friedrich-Schiller-University of Jena, Germany. Its major focus is not on the coupling of existing, complex environmental models but rather on the creation of problem-tailored models from well-defined, scientifically sound process simulation components. As an example, these components simulate interception, potential evapotranspiration or soil temperature based on physically based algorithms. Depending on the model’s purpose and available input data, a user can choose from the available process implementations that best matches her or his demands. But JAMS is not just a framework for component coupling, but also contains tools and graphical user frontends that support all stages of a model’s design and application lifecycle, like configuration, calibration, sensitivity and uncertainty analysis, and the exploration and visualization of modelling results.

Using JAMS, scientists can implement their expert knowledge as simulation components, even if little information about the later application context might be available at this stage (e.g. other components they might be coupled with). While providing a rich set of features useful for the application of simulation models (e.g. saving a models state during simulation for later continuation or providing server based and parallel execution environments), the framework’s invasiveness is minimized as

much as possible in order to ensure a fast learning process. Process components implemented for JAMS and models constructed from them mainly focus on the simulation of the hydrological and nutrient cycle (Kralisch et al., 2007).

The next section will give an overview of the JAMS architecture, its main concepts, model building blocks and dynamic runtime behaviour. Section 3 is dedicated to introduce various JAMS support tools covering the design and deployment of models as well as their calibration and analysis. Finally, section 4 will give a short overview of currently available JAMS-models and list some example applications.

2 System description

2.1 Framework overviewThe JAMS framework is structured into three main sections, i.e. (i) the core library, (ii) the runtime system, and (iii) the base component repository. The core library contains the API definition and different global support functions, e.g. for loading models or model data I/O. The application of models is managed by the runtime system, which is responsible for loading and executing JAMS models and provides additional services for event logging or profiling. The base component repository offers standard functions often used in environmental simulation models, e.g. providing geospatial processing capabilities. These central sections are complemented by different support tools and a graphical user frontends that are described in section 3. Interactions and dependencies between JAMS elements are depicted in Figure 1. The core library provides interfaces and data types both for the creation of modeling components and the runtime system. The latter can create and execute a JAMS model by accessing the component repository and a model definition, which is provided as a XML document. The creation of model definitions is supported by graphical user frontends covering the selection, linkage and parameterization of modeling components.

Figure 1: JAMS framework elements

2.2 Model building blocksThe main building block of any JAMS model is named component. A component is a JAVA class that implements three different methods (i.e. init, run and cleanup) that are triggered at according runtime stages that a JAMS model iterates through. As indicated by their name, these methods are executed at the beginning, in-between and at the end of a component’s lifetime. While the init and clean methods are executed only once, the run method can be invocated multiple times depending on the component’s purpose. Communication with the framework and other components is handled by arbitrary public attributes that fulfil two conditions: (i) they are of a valid JAMS data type and (ii) they

are marked by a special JAVA annotation, i.e. a syntactic meta-information. Attribute annotations are used for defining their I/O type (read, write), their default values, physical unit and boundaries (if numeric), and a text describing the attribute’s purpose. This information is used both by the runtime system to setup and interlink attributes and by frontend tools to give support during model design. As an example, a component could calculate potential evapotranspiration (pET), taking wind speed, temperature, humidity, radiation and elevation as input and calculating pET as output.

Components are stateless objects, meaning that all processing information has to be provided via attributes at each invocation of the component’s run method. In the case that a spatial state attribute (e.g. a water storage) should be modified by a component, the according value must be presented as an input and provided as an output of the component. The storage of the value till the next invocation of the component is done outside.

Although theoretically possible, temporal and spatial iteration is not meant to be done inside of components. Instead, this is subject of the second type of model building blocks, called contexts. A JAMS context is a special component that can nest other components and contexts, named children. A context serves two main purposes: (i) it controls the execution of its children, and (ii) serves as an application context for them. Depending on the purpose of a given context, it might invoke its children multiple times over a number of iterations or only once if some predefined condition is satisfied (Figure 2).

Figure 2: Context examples

Examples for iterator contexts are the spatial context or the temporal context, iterating over a set of spatial model entities or time steps respectively and invoking their children for each of them (see section 2.4). A typical application of both contexts is shown in Figure 3, with a spatial context nested inside of a temporal context. Using this setup, conceptual models that simulate environmental processes at discrete points in space-time can easily be represented.

For spatial contexts, different sets of spatial attributes (each of them representing one spatial model entity) must be stored after each iteration step and restored prior to the next iteration step of the same entity. This task is managed by state objects that are associated to contexts, each of them storing a set of attributes and their values. Depending on whether a context must store its iteration states or not, it can have multiple states (spatial context) or only one (temporal context).

Children iteration is done in a context’s run method which it inherits from the component class (Figure4).

Figure 3: Nesting of temporal and spatial contexts

Although contexts are stateless like components, they can manage a set of special states. A state is characterized by a dynamic set of attributes, reflecting its components’ inputs and outputs. As a context might iterate over a series of children invocations, one state object might exist for each iteration. The relation between components, contexts and states is shown in Figure 4.

, serving as a data proxy for their child are stateful onkecstore their state after Details of the data

stage are used to implement specific knowledge as process algorithms whereas the model structure and the components’ execution control are defined by the context components. As a benefit from this approach, an environmental process (e.g. potential evapotranspiration) can be implemented as a JAMS component without any knowledge about its later execution context, e.g. the temporal resolution or the type of spatial discretisation of the modelled area. The only precondition for the later application of a component in conjunction with others is the proper declaration of its desired input data and provided output data by means of metadata in the component's source code. Context components in JAMS define the model structure that is needed to represent the necessary temporal and spatial iteration loops. This can be accomplished by managing the repeated execution of other components while varying their sets of input data. Each set of input data may then represent one point in time ( temporal context) or space (spatial context). A context component in JAMS can thus be seen as a scope that defines an environment for the execution of other software components. Together with another context (model context) that manages the execution of the real simulation, the above-mentioned environmental models can easily be produced.

Figure 4: Components, contexts and states

As JAMS is a JAVA-based framework, components and contexts are implemented as JAVA classes. For ease of use, newly implemented components can be inherited from existing base classes that already provide the necessary interfaces. In order to offer meta-information about components or contexts (e.g. for later use by the runtime system), JAVA annotations are used. They are

2.3 Dynamic behavior

2.3.1 Run stages

2.3.2 Data exchange

High level model design

Discrete time/space system

Timed event system

Non-linear

Non-autonomous

A JAMS simulation component is a software component which is characterized by a tuple (I, O, P) where I is a set of input

In contrast to Discrete Event Systems, state transitions of JAMS simulation components are not autonomously triggered by external events. Instead, state transitions and thus the calculation of new output values are explicitly triggered

Stateful / stateless context

A context’s behavior defines a sequence of states which is traversed during the context’s run stage. A state can be understood as a dynamic, user defined set of attributes. For each context state all of the context’s child components are processed by executing their run method. As the child components input and output slots can be linked to any enclosing context’s state attributes, a context state can be seen as a data proxy that stores a component’s result beyond the components life span.

, each of them characterized by a distinct set of attribute values. These attributes can be modified

2.3.3 Data exchange

2.4 Core components

3 Support tools

3.1 Model design and deployment

3.2 Parameter optimization and analysis

3.2.1 Sensitivity analysis and calibration

The JAMS-Explorer is offering a powerful toolbox for sensitivity analysis, uncertainty estimation and model identification and visualization based on the Monte Carlo Toolbox Version 5 (MCAT5) developed by Wagener et al. (2007). Although MCAT was designed to meet requirements from the domain of hydrologic modeling, it is suited for most mathematical models as well. The following section provides firstly an overview about data requirements of the toolbox and the associated data-scheme. Secondly, all implemented methods are listed and a selection of three tools is explained in more detail.

Most of the methods included in the toolbox require data from a prior Monte Carlo sampling. Monte Carlo sampling relies on repeated simulation of a model with randomly generated or perturbed parameter or input-data. To perform this task, the JAMS Framework includes some random sampler components. Section 4.2.2 gives further information about random sampling. The result from this sampling is the compilation (ensemble) of all the model responses consisting of model-parameters, one or more objectives and some spatial and time series data. Here, objective is specified as any single valued function of the model response, but in almost all cases a goodness-of-fit measure between simulated and observed data is supposed.

Because of the huge amount of data, processing of ensembles usually requires a lot of computing power and even more memory capacity. The general concept of JAMS’ data analysis and processing is not suited for this task. Therefore, the demand for a special data-scheme arises, which meets the following requirements:

- Efficient data-storage with regard to memory consumption- Fast and simple creation of ensembles from raw data- Flexibility and extensibility in respect to data-types used in the data-scheme- Incorporation of metadata

The core of the resulting scheme consists of four classes i) DataSet, ii) Simulation, iii) DataCollection and iv) Ensemble. The root of the scheme is the class DataSet. Therefore each class of the scheme has to be an instance of DataSet. The class Simulation is responsible for storing and enumerating all DataSets associated with a single model simulation. These DataSets are categorized by their name and class. Simulations also containing some metadata and ensure basic data consistency, but for the sake of flexibility they are neither intended to know much about their data nor offering methods for data manipulation. Here the class DataCollection comes into play, which is a powerful container for Simulations. It provides essential operations to search and manipulate DataSets and it is able to work with ensembles. Another feature of DataCollection is creation of Ensembles by simply providing properties of the desired DataSets. Then all DataSets which meet the requirements are retrieved from the collection of simulations and compiled into a new ensemble. Furthermore it is possible to merge multiple DataCollections into a new one, even when there are different temporal extends or different DataSets. The framework is able to handle such request and ensures some degree of data-consistency. Figure 5 shows the structure of the scheme and some of the provided operations.

Figure 5 Data scheme in UML notation

Method Application

Dotty Plot Sensitivity Analysis, Model identificationA Posteriori Parameter Estimation Model identificationIdentifiability Plot Model identificationDYNIA Temporal sensitivity analysis of parametersRegional Sensitivity Analysis Sensitivity analysisRegional Sensitivity Analysis II Sensitivity analysisClass Plot Comparison of behavioral and non-behavioral modelsGLUE Variable Uncertainty Uncertainty estimation of one-valued model outputsGLUE Output Uncertainty Uncertainty estimation of time-series generated by a model

Pareto Output UncertaintyUncertainty estimation of time-series generated by a model based on Pareto estimation of multiple objectives

Multi-Objective Plots Evaluation of dependencies between different objectives

Normalized Parameter Range PlotsVisualization of the best parameter values with respect to different objectives.

Best Prediction PlotsVisualization of the best simulation with respect to different objectives

Table 1 MCAT methods available in JAMS–Explorer for Monte Carlo simulation analysis

The next section describes three common challenges that can be addressed with MCAT.

3.2.1.1 Model identificationA model is seen as identifiable iff, all the behavioral parameterizations of the model are located in a relative small region of the whole parameter-space. Here we assume that a behavioral parameterization is defined as parameter combination with plausible model response. With other words a model is identifiable if the behavior of the real system is explained only by one kind of parameterization. If a model is identifiable it can directly be used to draw conclusions about the hydrological system, e.g. assign a real physical meaning to some model parameters or using the model for predictions or scenario modeling. It can also provide some deep insight into the hydrological system.

A Dotty Plot is special scatter plot for assessing identifiability. Each point represents the value of an objective or variable achieved with a specific parameter combination. To visualize the data the whole parameter space is

projected in one dimension by selecting one parameter and ignoring the others. The surface or upper boundary line of the dots represents an estimate for the best model realization achievable for a given value of the selected parameter. If there is a well-defined maximum then the selected parameter could be considered to be well identified. Figure 6 shows the difference in the Dotty Plot between an identifiable parameter and a parameter which is not identifiable.

It should be mentioned that many models are not or only partly identifiable, i.e. there are very different parameterizations explaining the system well. This is known as Equifinality. The reason for this is that different internal states of a model can result in similar model responses. It is not often feasible to validate internal model states. Because it is impossible to say which model is correct, no model could be refused. As a logical consequence an ensemble simulation should occur with all possible candidates. This functionality is also supported by JAMS.

Figure 6 the left figure shows a parameter which could easily identified, where the parameter in the right picture is not identifiable.

3.2.1.2 Sensitivity AnalysisA wide range of applications require identification of dominant parameters and processes of a hydrological model by performing a sensitivity analysis.

The Regional Sensitivity Analysis Tool (Spear and Hornberger, 1980) is a method for estimating sensitivity of model parameters, where sensitivity is defined as effect on the entire model performance. This method splits the model simulations into ten groups according to their model performance. Thus the first group consists of the most behavioral simulations (best ten percent) and the last group consists of the most non-behavioral simulations (worst ten percent). For a given parameter, the accumulated distribution of the simulations across the valid range of the parameter is calculated. This is carried out separately for each group. Afterwards each distribution is visualized in a combined plot. A large discrepancy between those distributions indicates parameter-sensitivity where similarity between the distributions indicates weak sensitivity. Figure 7 shows an example for a sensitive and non-sensitive parameter.

Applications of sensitivity analysis range from questions of decision support to hydrological – system – analysis. Another field of application is model calibration. It is known, that the effort to calibrate a model increases highly with the number of parameters. Hence it is not recommended to calibrate all model-parameters at once, but to split the calibration process into several subtasks by taking into account only a subset of the parameters. With this approach it is advisable to begin with the most sensitive parameters, because they are also most important in terms of calibration. A further improvement is to identify all identifiable parameters and to assign them to their optimal value. For both tasks MCAT-5 provides the necessary tools.

3.2.1.3 Uncertainty EstimationIt is very unusual for hydrological models to exactly simulate reality. Uncertainties in the model structure, input-data and parameterization leads to some uncertainties in the model response. It is very important to quantify these uncertainties to derive robust predictions and conclusions from a simulation. Uncertainty estimation in JAMS-Explorer is based on the Generalized Likelihood Uncertainty Estimation (GLUE) Framework, which uses Monte – Carlo – Sampling and disturb input-data and parameterization with random errors. This gives an estimate about the effect of real measurement errors and parameter-uncertainty. The basic idea behind GLUE is that the likelihood of simulation is reality is directly related to the value of an objective. Therefore the objective values are linear transformed into a probability distribution. In a preprocessing step non-behavioral simulations are removed from the ensemble. Then the probability distribution is used to generate confidence intervals.Figure8 shows the confidence intervals generated from parameter uncertainty with the GLUE Uncertainty Estimation Tool.

Figure 7 Regional Sensitivity Analysis: The left plot shows a very sensitive parameter. The right plot shows an insensitive parameter.

3.2.1.4 CalibrationIn order to apply environmental models successfully, it is essential to determine model parameters very carefully. A direct determination of parameter values is often not feasible since they lack a clear physical meaning or field measurements are too expensive. Therefore, estimation of parameters is usually done by fitting model response and observation in a trial and error process. Depending on the model's complexity and the number of calibration parameters, this process can often be very time-consuming.

Model calibration with JAMS is supported by some powerful tools for optimization, which have been implemented as JAMS context components. In various studies it was shown that the available algorithms are able to find good and best parameterizations automatically. A noticeable advantage of JAMS is that due to the flexibility of the JAMS framework and the generality of the applied methods, only a minimum of knowledge about the model to be calibrated is needed. Only the relevant model parameters, the objective function(s) and method dependent parameters must be made available to the calibration component.

The process of calibration with JAMS consists of several steps as shown in Error: Referencesource not found.

i) At first the optimization context is initialized, e.g. optimization parameters are read and variables are initialized. ii) The next iteration is started with generation of a new parameter-set p, according to the rules of the optimization method. iii) With this parameter-set p a model simulation is carried out. iv) The objective is evaluated. The optimization routines do not contain any functionality for calculating objective values, but the JAMS –

Framework is offering some components for model assessment e.g. calculation of the Nash-Sutcliff-Efficiency (E2 ¿, Coefficient of determination (r2 ) or absolute volume-error ( AVE). To pass the calculated objective to the optimizer the functionality of JAMS – Attributes is used. It is possible to optimize a single objective or multiple objectives at a time or simply performing a random sampling for Monte-Carlo-Analysis. There are significant differences between those options, which are summarized in Table 2. v) Steps ii-v is reiterated until a stopping criterion is fulfilled.

Random Sampling Single objective optimization Multi objective optimization

Formal description of the set of solutions

L=~p⊆P ,|~p|=k L={x : f ( x )=minp∈P

f ( p )} L= {x :∀ y∈ P∃i=1. .m f i ( x ) ≤ f i ( y ) }

Figure 8 0.95 Confidence intervals, calculated with GLUE uncertainty estimation. Nine different parameters of the J2000 were disturbed during a Monte - Carlo - Analysis

Number of objectives - 1 m

Description of set of solutions

Any arbitrary collection of parameter-sets (some) global optimal point(s) Pareto – Front

Example of a solution

Solvers in JAMS

RandomSamper Latin Hyper Cube

Random Sampler

Shuffeld Complex Evolution

DIRECT Simple Branch & Bound Gutmann Method Nelder Mead

MOCOM NSGA-II MOSCEM

Example

Random Samplers are performing simply a fixed number of model simulations. To perform multiple simulations with random input error the InputError-Estimation component could be used, which adds an absolute or relative error to measurement data.

DIRECT is a typical deterministic optimization routine based on the Branch&Bound paradigm. The class of Branch & Bound algorithms has a long history in mathematical optimiz-ation. The common strategy of these methods is to partition the problem into smaller sub problems (Branch), choose one of the sub problems and recursively apply this strategy. As soon as all sub problems have been solved, the solution of the original problem is also known. To speed up this process sub problems are rejected if they are unlikely to contain a global solution (Bound). A detailed explan-ation of this algorithm is given by Horst et al. (2000).

NSGA-2 is famous multi-objective optimization routine, which has been attracted a lot of interest in the community of mathematical optimization. NSGA-2 has shown good performance in many studies (Bui et al. 2004, Coello & Sierra, 2003, Zitzler et al., 2000). The algorithm belongs to the class of genetic algorithms, which is a subset of evolutionary algorithms.Solution candidates in evolutionary algorithms are seen as population of individuals. The process of optimization develops this population by creating new generations with common operations like selection, mutation and crossover, which are inspired by biological evolution.

Table 2 Properties of single and multiple objective optimizations

3.2.1.5 OPTAS

For the calibration of JAMS-models, various adjustments of the model structure are desirable, e.g. selection, insertion and parameterization of the optimization component, removal of visual output components and selection of in- and output data. The software wizard OPTAS provides support in this task and guides the user step by step through the process of calibration and gives assistance whenever possible.

As a first step of the calibration setup procedure, the model is loaded and the list of available parameters is presented to the user. Following, feasible value ranges of the selected parameters and one or more objective functions (e.g. Nash-Sutcliffe efficiency) need to be selected from a list. Based on this information, suitable optimization methods are proposed and parameterized with default values. For fine tuning, these parameterizations can be adapted. A comprehensive description of all values helps to choose well-performing parameters. Finally the software wizard automatically generates the calibration environment.

In order to maximize runtime performance of the model and accordingly of the calibration procedure, all components that are not needed to ultimately compute the objective function can be removed. In order to identify

those components, a dependency analysis of the model is being applied based on the components’ input and output data.

Based on the resulting dependency-graph the model structure can be further analyzed. A component which is not connected with the objective function component does neither directly nor indirectly use any data generated by that component. That means such a component has absolute no effect on the objective and is therefore eliminated. Because every component provides information about their attributes this analysis is done fully automatically. Usually, these are components for visualization and data-output. In addition, components can be identified that have influence on the objective function, but are independent of the parameters in calibration, e.g. data-input processes. Under specific circumstances, these processes can be executed only once before calibration starts, thereby saving execution time. In order to make sure that each model starts with identical initial conditions, a snapshot of the complete model is being taken. At each model execution in the course of the optimization procedure, this snapshot is being restored in advance.

4 ApplicationsSimulation components and models available

5 SummaryJAMS is developed as open source software licensed under the GNU Lesser General Public License and can be downloaded from http://jams.uni-jena.de. The package includes the JAMS core system, its GUI extensions for model creation and execution, a library with frequently used components (e.g. for creating plots and maps from simulation results), and the J2000 hydrological modelling components. In order to test JAMS, the package also provides a preconfigured, ready-to-run J2000 model with test data.

6 BibliographyArgent, R., Perraud, J., Rahman, J., Grayson, R., & Podger, G. (2009). A new approach to water quality modelling and environmental decision support systems. Environmental Modelling & Software, 24(7), 809-818.

Argent, R., Voinov, A., Maxwell, T., Cuddy, S., Rahman, J., Seaton, S., Vertessy, R., et al. (2006). Comparing modelling frameworks – A workshop approach. Environmental Modelling & Software, 21(7), 895-910.

Branger, F., Braud, I., Debionne, S., Viallet, P., Dehotin, J., Henine, H., Nedelec, Y., et al. (2010). Towards multi-scale integrated hydrological models using the LIQUID® framework. Overview of the concepts and first application examples. Environmental Modelling & Software, 25(12), 1672-1681.

Bui, L.T. and Essam, D. and Abbass, H.A. and Green, D.(2004). Performance analysis of evolutionary multi-objective optimization methods in noisy environments. Proceedings of the 8th Asia Pacific symposium on intelligent and evolutionary systems 29-39

Coello, C. and Sierra, M.R.(2003). A coevolutionary multi-objective evolutionary algorithm. IEEE Congress on Evolutionary Computation, 2003. CEC'03. 482-489

David, O., Markstrom, S. L., Rojas, K. W., Ahuja, L. R., & Schneider, I. W. (2002). The Object Modeling System. In L. Ahuja, L. Ma, & T. A. Howell (Eds.), Agricultural System Models in Field Research and Technology Transfer (pp. 317–31). Lewis Publishers.

Fink, M., Krause, P., Kralisch, S., Bende-Michl, U., & Flügel, W. (2007). Development and application of the modelling system J2000-S for the EU-water framework directive. Advances in Geosciences, 11, 123-130.

Fischer, C., Kralisch, S., Krause, P., Fink, M., & Flügel, W. (2009). Calibration of hydrological model parameters with the JAMS framework. In R. Anderssen, R. Braddock, & L. Newham (Eds.), Proceedings of the 18th World IMACS Congress and MODSIM09 International Congress on Modelling and Simulation (pp. 866-872). Cairns, Australia.

Gregersen, J., Gijsbers, P., & Westen, S. (2007). OpenMI: Open modelling Interface. Journal of Hydroinformatics, 3(9), 175–191.

Kralisch, S., Zander, F., & Krause, P. (2009). Coupling the RBIS Environmental Information System and the JAMS Modelling Framework. In R. Anderssen, R. Braddock, & L. Newham (Eds.), Proceedings of the 18th World IMACS Congress and MODSIM09 International Congress on Modelling and Simulation (pp. 902-908). Cairns, Australia.

Kralisch, S., Krause, P., Fink, M., Fischer, C., & Flügel, W. (2007). Component based environmental modelling using the JAMS framework. In D. Kulasiri & L. Oxley (Eds.), Proceedings of the MODSIM 2007 International Congress on Modelling and Simulation. Christchurch, New Zealand.

Krause, P., Bende-Michl, U., Fink, M., Helmschrot, J., Kralisch, S., & Künne, A. (2009). Parameter sensitivity analysis of the JAMS/J2000-S model to improve water and nutrient transport process simulation - a case study for the Duck catchment in Tasmania. In R. Anderssen, R. Braddock, & L. Newham (Eds.), Proceedings of the 18th World IMACS Congress and MODSIM09 International Congress on Modelling and Simulation (pp. 3179-3186). Cairns, Australia.

Krause, P., Kralisch, S., & Flügel, W. (Eds.). (2005). Model Integration and Development of Modular Modelling Systems. Advances in Geosciences (Vol. 4). European Geosciences Union.

Krause, P. (2002). Quantifying the impact of land use changes on the water balance of large catchments using the J2000 model. Physics and Chemistry of the Earth, 27(9-10), 663-673.

Rahman, J. (2004). Making frameworks more useable: using model introspection and metadata to develop model processing tools. Environmental Modelling & Software, 19(3), 275-284.

Rahman, J. M., Seaton, S. P., Perraud, J. M., Hotham, H., Verrelli, D. I., & Coleman, J. R. (2003). It's TIME for a new environmental modelling framework. In MODSIM 2003: International Congress on Modelling and Simulation (pp. 1727-1732). Nedlands, WA, Australia: Univ Western Australia.

Spear, R.C., Hornberger, G.M. (1980). Eutrophication in Peel Inlet, 2, Identification of critical uncertainties via Generalised Sensitivity Analysis. Water Research, 14, 43-49

Wagener, T., & Kollat, J. (2007). Numerical and visual evaluation of hydrological and environmental models using the Monte Carlo analysis toolbox. Environmental Modelling & Software, 22(7), 1021-1033.

Dias, A.H.F. & De Vasconcelos, J.A.(2002). Multiobjective genetic algorithms applied to solve optimization problems. IEEE Transactions on Magnetics, 38(2), 1133-1136.

CCA

HLA

OpenMI

MMS

OMS

TIME

LIQUID

Jagers, H.R.A. (Bert), 2010. Linking Data, Models and Tools: An Overview, in: Swayne, D.A., Yang, W., Voinov, A.A., Rizzoli, A., Filatova, T. (Eds.), Proceedings of the iEMSs Firth Biennial Meeting: International Congress on Environmental Modelling and Software (iEMSs 2010). International Environmental Modelling and Software Society, Ottawa, Canada.