Verification and Validation of Agent-based Scientific

Simulations

1

Xiaorong Xiang, Ryan Kennedy, Gregory MadeyComputer Science and Engineering

University of Notre Dame

Steve CabanissDepartment of ChemistryUniversity of New Mexico

2

OverviewIntroduction

Concepts of Verification and ValidationResearch Objectives and Methods

A Case StudyApply Verification and Validation Methods to the Case StudyConclusionFuture Work

3

Model Verification & Validation (V & V)V & V

Verification: get model rightValidation: get right model

The cost and value influence confidence of model acceptance level

*Adapted from Sargent: “Verification andValidation of Simulation Models”

4

V & V for Agent-based SimulationAgent-based modeling is a new approachDifferent than Queuing Models

Entities: large number of heterogeneous active objects vs. passive objects

Space: continuous or discrete grid space vs. network of servers and queues

Interactivity: high vs. lowActive components: agents vs. queues and serversGoal: discovery vs. design and optimization

Few literature to date address the formalized methodology for V & V of Agent-based Simulations

5

What and HowResearch objective

Generate guidelines or a formalized methodology for V & V of Agent-based Simulations

HowNOM project as a case studyEvaluate and adapt the formalized V & V techniques in industrial and system engineering for DESIdentify a subset of these techniques that are more cost-effective for Agent-based Simulations

6

NOM Agent-based Simulation ModelNSF funded interdisciplinary project

Understanding the evolution and heterogeneous structure of Natural Organic Matter (NOM)E-science exampleChemists, biologists, ecologists, and computer scientists

Agent-based stochastic model Web-based simulation model

7

NOMWhat is NOM?

Heterogeneous mixture of molecules in terrestrial and aquatic ecosystems

Why study NOM?Plays a crucial role in the evolution of soils, the transport of pollutants, and the global carbon cycleUnderstanding NOM helps us better understand natural ecosystems

8

The Conceptual Model IAgents

A large number of moleculesHeterogeneous properties

Elemental compositionMolecular weightCharacteristic functional groups

BehaviorsTransport through soil pores (spatial mobility)Chemical reactions: first order and second orderSorption

9

Stochastic Synthesis: Data Model

Pseudo-Molecule

Elemental Functional Structural

Composition

CalculatedChemicalProperties

and Reactivity

LocationOriginState

10

The Conceptual Model IIStochastic Model

Individual behaviors and interactions are stochastically determined by:

Internal attributesMolecular structureState (adsorbed, desorbed, reacted, etc.)

External conditionsEnvironment (pH, light intensity, etc.)Proximity to other molecules

Length of time step, ∆t

Space2D Grid Structure

Emergent propertiesDistribution of molecular properties over time

11

Implementations

12

V & V of the NOM ModelExamples of V & V techniques

Face validityAnimationGraphical representation

TracingInternal validityHistorical data validation (calibration sets and test sets)Sensitivity analysisPrediction validationComparison with other modelsTuring test

13

V & V of NOM Simulation Model

*Adapted from Sargent: “Verification and Validation of Simulation Models”

14

Face Validity

15

Internal Validity I

0

500

1000

1500

2000

2500

3000

3500

4000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Seed

Num

ber

of M

olec

ules

at t

he e

nd s

tep

AfterBefore

16

Internal Validity II

Number of Molecules

1775.0

1725.0

1675.0

1625.0

1575.0

1525.0

1475.0

1425.0

1375.0

1325.0

1275.0

1225.0

Freq

uenc

y60

50

40

30

20

10

0

Std. Dev = 100.79 Mean = 1473.9

N = 450.00

17

Model-to-Model Comparison ICompare the model with validated oneCompare the model with non-validated oneDifferent implementations

Different programming languagesDifferent packages

Different modeling approachesPredator-Prey model

Agent-based approach vs. System Dynamics approach

Powerful method for ABS

18

Model-to-Model Comparison IIFeatures Alpha Step No-flow ReactionDeveloping Group University of New

Mexico, chemistsUniversity of Notre Dame, computer scientists

Programming language Pascal Java (Sun JDK 1.4.2)

Platforms Delphi 6, Windows Red hat Linux cluster

Running mode Standalone Web based, standalone

Simulation package None Swarm, Repast libraries

Animation None Yes

Spatial representation None 2D grid

Second order reaction Random pick one from list

Choose the nearest neighbor

First order with split Add to list Find empty cell nearby

19

Model-to-Model Comparison III

20

Model-to-Model Comparison IV

21

Model-to-Model Comparison V

22

Conclusion and Future WorkV & V Case StudyModel-to-Model Comparison is Powerful

Collect and evaluate more statistical dataCompare simulation results against empirical dataTweak V & V methodsGenerate guidelines and methodology for V & V of agent-based simulation models

Questions or Comments?

23