TOOLS FOR STATISTICAL AND BEHAVIORAL ANALYSIS OF MASON MODELS

László Gulyás, Richard O. Legendi, Tamás Máhr

Rajmund Bocsi, Vilmos Kozma, Peter Rieger{lgulyas, rlegendi, tmahr, rbocsi, vkozma, prieger}@aitia.ai

AITIA International, Inc. – Intelligent Applications and Web Services

NSF MASON WorkshopGeorge Mason University

15-16 June, 2013, Fairfax, USAhttp://cs.gmu.edu/~eclab/projects/mason/workshop/pmwiki.php

The research leading to these results has received funding from the European Union, Seventh Framework Programme FP7/2007-2013 under grant agreement no. CRISIS-ICT-2011-288501.

NSF MASON Workshop 2

DEMO

2013.06.15.

NSF MASON Workshop 3

The El Farol Bar Problem• Agents are researchers (N=100)• They visit a popular but small bar in Santa Fé

• If attendance > 60 (overcrowded)• Who hasn’t come

• If attendance <= 60 • Who hasn’t come

• Each day agents decide individually and in the same time

W. Brian Arthur, “Inductive Reasoning and Bounded Rationality”, American Economic Review (Papers and Proceedings), 84,406–411, 1994.

2013.06.15.

NSF MASON Workshop 4

The El Farol Bar Problem• MASON implementation of a NetLogo variant• Artifical agents:

• ARMA-based prediction with history

• Players have two actions: No go / Go

• If +1 Score!• Goal: get max score

Rand, W. and Wilensky, U. (2007). NetLogo El Farol model. Center for Connected Learning and Computer-Based Modeling, Northwestern Institute on Complex Systems, Northwestern University, Evanston, IL. http://ccl.northwestern.edu/netlogo/models/ElFarol

2013.06.15.

NSF MASON Workshop 5

Let’s play!

http://demo1.aitia.ai/

Simulations

2013.06.15.

NSF MASON Workshop 62013.06.15.

Model List

Create Experiments

Additional Info

NSF MASON Workshop 7

Admin Page (Admins are subjects too)

2013.06.15.

Subject list(joined)

Admin tools

NSF MASON Workshop 8

In-experiment Admin Page

2013.06.15.

Player status (moved/waiting)

NSF MASON Workshop 9

In-experiment Subject Page

2013.06.15.

Status messages

NSF MASON Workshop 10

Post-experiment Scores

2013.06.15.

NSF MASON Workshop 11

Outline• Background

• The CRISIS project

• CRISIS Simulation Library• CRISIS Game & PET v2.0• MEME• Future works

2013.06.15.

NSF MASON Workshop 12

Background: The CRISIS Project“The CRISIS project addresses building a next generation macroeconomic and financial system policymaking model: a bottom-up agent-based simulation that fully accounts for the heterogeneity of households, firms, and government actors. The model will incorporate the latest evidence from behavioral economics in portraying agent behavior, and the CRISIS team will also collect new data on agent decision making using experimental economics techniques. While any model must make simplifying assumptions about human behavior, the CRISIS model will be significantly more realistic in its portrayal of relevant agent behavior than the current generation of policymaking models.”

Crisis project description: https://www.crisis-economics.eu/

2013.06.15.

NSF MASON Workshop 13

CRISIS Simulation Library• Olaf covered that

2013.06.15.

NSF MASON Workshop 142013.06.15.

Web-based Game(Participatory Experiments)

Economic Simulator(Cloud-Based Parameter

Sweep Execution)

ModellingFramework

Models

NSF MASON Workshop 152013.06.15.

Web-based Game(Participatory Experiments)

Economic Simulator(Cloud-Based Parameter

Sweep Execution)

NSF MASON Workshop 16

CRISIS Game

2013.06.15.

(Planned design)

http://game.crisis-economics.eu

NSF MASON Workshop 17

The Model Under the CRISIS Game

2013.06.15.

Source: Domenico Delli Gatti, personal communications

(Single player mode)

Domenico Delli Gatti, Saul Desiderio, Edoardo Gaffeo, Pasquale Cirillo, and Mauro Gallegati. Macroeconomics from the Bottom-up. 1st ed. Springer, 2011.

NSF MASON Workshop 18

Architecture

2013.06.15.

NSF MASON Workshop 19

PET v2.0• Idea came from AITIA's PET Software

• Participatory Extension

• Generalized components behind the CRISIS Game• Modern, scalable infrastructure

• Previous version is 5+ years old • Components may run on (several) different machines

• Software already available for any experiment

2013.06.15.

NSF MASON Workshop 20

PET v2.0• Can be used:

• Locally• Laboratory experiments• Policy makers (scenario analysis with a proper model)

• On any webserver to run constantly• Gather data (scores from model and all user actions replay)• Dissemination

• Questionnaire module• Verify if subjects understood the rules

• Software is already in use by Universiteit van Amsterdam

2013.06.15.

NSF MASON Workshop 21

What is required to use?

1. An ABM model• PET is a generic framework• Currently Java and MASON models are supported• (But! No serialization is required )

2013.06.15.

NSF MASON Workshop 22

What is required to use?

1. An ABM model• PET is a generic framework• Currently Java and MASON models are supported• (But! No serialization is required )

2. Development of the web interface• No restriction on tools (HTML5, GWT, ...)• Communication to server by standardized tools (AJAX)

• Config files to map incoming messages to model actions

2013.06.15.

NSF MASON Workshop 23

MEME

2013.06.15.

WEHIA 2012 - Replication of MacroABM 24

Parameter sweeps?

„[...] suppose that in a model there are just 10 relevant parameters, and that each parameter can assume 10 different values (a rather simplifying assumption). As a result, one obtains that the constellation of the parameter space is given by 10^10 vectors. If we perform 20 different runs for each one of them to take into account the possible effects of changing the random seeds, the total number of simulations would amount to 2*10^11!”

2012.06.22.

Domenico Delli Gatti, Saul Desiderio, Edoardo Gaffeo, Pasquale Cirillo, and Mauro Gallegati:Macroeconomics from the Bottom-up. 1st ed. Springer, 2011 (p. 76., section 3.10.1)

WEHIA 2012 - Replication of MacroABM 25

Parameter sweeps?

„[...] suppose that in a model there are just 10 relevant parameters, and that each parameter can assume 10 different values (a rather simplifying assumption). As a result, one obtains that the constellation of the parameter space is given by 10^10 vectors. If we perform 20 different runs for each one of them to take into account the possible effects of changing the random seeds, the total number of simulations would amount to 2*10^11!”

2012.06.22.

Domenico Delli Gatti, Saul Desiderio, Edoardo Gaffeo, Pasquale Cirillo, and Mauro Gallegati:Macroeconomics from the Bottom-up. 1st ed. Springer, 2011 (p. 76., section 3.10.1)

Even if 1 run / second would take ~6 585 000 years

NSF MASON Workshop 26

MEME – Model Exploration Module• MEME is the tool for the job

• Standalone application• Parameter sweep exploration

• Being Strong• Exploiting Grid/Cloud systems

(LAN, Amazon, etc.)

• Being Smart• Design of Experiments

2013.06.15.

NSF MASON Workshop 27

MEME – Model Exploration Module• Generic tool

• Supports various platforms (NetLogo/Repast J/MASON/...)

• Easy-to-use graphical interface• Setting the parameter tree• Recording• External tool support (R/Matlab/Octave/...)

• Recently declared an open source project

2013.06.15.

NSF MASON Workshop 28

Design of Experiments

• Reduce # of runs• Ignore unimportant cases

„smartly”• Factorial Designs (2/3 lev.)• Latin Hypercube,• Box-Behnken,• Box-Wilson Central

Composite • Genetic Algorithm Designs,• Iterative Uniform

Interpolation• Adaptive Nonlinear Tests

2013.06.15.

NSF MASON Workshop 29

Demo

2013.06.15.

NSF MASON Workshop 30

FUTURE WORKS

2013.06.15.

NSF MASON Workshop 31

Future Works• Pet v2.0

• Tools to support front-end creation• JavaScript library (WIP)• User-friendly Eclipse editor plug-ins (flexible, easy to use)

• Both for the UI, questionnaire

• Tutorials, docs to be developed• Additional platforms

• MEME• Publish open source project, documentation...

• Cloud-based economic simulator

• +Your ideas?

2013.06.15.

NSF MASON Workshop 32

Interested?

• „A New Set of Tools Supporting Agent-Based Economic Modeling”, EPIA 2013 (under publication, ~Sept.)

• Statistical and Behavioral Model Analysis Tools.• Hands-on workshop @ ESSA 2013

• CRISIS Website: http://www.crisis-economics.eu/

2013.06.15.

NSF MASON Workshop 33

THANK YOU!• Questions?

• Will be available soon from http://iaws.aitia.ai/

• Download the CRISIS Simulation Library:• http://www.crisis-economics.eu/software

• Try the CRISIS Game• http://game.crisis-economics.eu

• Take a look on the El Farol model:• http://github.com/rlegendi/mason-examples

• Download MEME:• http://meme.aitia.ai

Richard O. Legendirlegendi@aitia.ai

http://people.inf.elte.hu/legendi/@legendi_ELTE

June 15, 2013.

2013.06.15.

The research leading to these results has received funding from the European Union, Seventh Framework Programme FP7/2007-2013 under grant agreement no. CRISIS-ICT-2011-288501.

NSF MASON Workshop 34

„Visionary things”• Usability:

• API level: making it easier to use (annotations, generics)• step(Object) vs. step(Model)

• DevTools:• Maven (Graddle, Ivy, Sbt)• Recording and parameterizing simulations

• Distribution• Networks• MASON Simulation Library for the community

2013.06.15.