1

Models, Simulations and Scientific Method

Caroline LyonSchool of Computer ScienceUniversity of Hertfordshire

http://homepages.feis.herts.ac.uk/~comrcml

29.11.2006

2

Overview

• What is “scientific method” ? - traditional approach - problem fitting models and simulations into the framework - necessary conditions for scientific simulations

Examples to illustrate criteria for evaluation of models and simulations• London underground map• Nowak’s model of the evolution of words in human language• Elman’s simulation of the development of grammar

• Conclusions: The difference between necessary and sufficient conditions for scientific method.

3

Scientific method – some historical landmarks

300BC Aristotle and methods of categorising knowledge

c. 1000AD Islamic mathematics and scientific experiments

1600 Bacon – controlled experiments, empirical scientific investigations

1637 Descartes - Discourse on Method, rationalism

1680 Newton – hypotheses that enabled predictions

1920 Fisher – statistical analysis

1934 Popper – falsifiability as a criterion

1962 Kuhn –The Structure of Scientific Revolutions

4

What is a model or simulation?

• Models and simulations contrast with real objects or events• Word model used in differing ways.

One end of a spectrum: an artifact which aims to reproduce the function of a real biological system as closely as possible. E.g. cochlear implant

A contrasting usage: language model, a term which denotes a collection of statistics on word frequencies

• Simulation typically means a model with dynamic elementsIEEE has 2000+ Computer Simulation Standards for scientific processesMDA – Model Driven Architecture

5

Characteristics of a scientific approach

• Objective versus subjective

• Both deductive ( as in mathematics) and empirical knowledge

• Supporting evidence, repeatable experiments– But limitations of an inductive approach e.g. all swans are white

• Production of hypotheses that can be falsified (cf Karl Popper)– Falsification rather than verification– Example of Newtonian mechanics– Contrast with other forms of belief, e.g.

“St Paul’s cathedral is a beautiful building”“It is wrong to eat people”

6

Problems fitting models into this framework

• Since models and simulations abstract certain features and ignore others they can be falsified by examining other features that were not abstracted.

• Example: the map of the London underground Purpose: to show passengers which route to take Functional efficiency: excellent But, not to scale. Relative distances between stations do not match actual

distances.

• Models and simulations cannot be assessed using a naïve falsification test

• Note that underground map is not arbitrary: corresponds topologically to the real system.

7

Map of London Underground

8

Feature abstraction (1) Picasso – portrait of a young girl

9

Feature abstraction (2) Picasso – portrait of a young girl

10

Criteria for evaluating simulations and models

• Consistency between empirical evidence and declared aims– fitness for purpose– external validity: a reality check

• Consistency within the model or simulation

– internal cohesion

• Well founded choice of parameters– arbitrary choices explicit– (caution in use of metaphorical language)

11

Use of simulations in language acquisition and evolution

Research into language acquisition - in evolutionary time- in historical time- in the lifetime of an individual

Simulations needed because• There are few ways in which we can find out about events millions of

years back. We create a virtual laboratory for experiments

• They avoid unethical investigations into the functioning of the brain

12

Example 1“Computational and evolutionary aspects of language” M. A. Nowak et

al. (Nature, Vol 417, 6.6.2002)

Aim: to produce a model, “a theoretical framework explaining how darwinian dynamics lead to fundamental properties of human language”

Process includes assumptions:“that a language can be seen as an infinite binary matrix linking phonetic forms to semantic forms”“ambiguity …. is the loss of communicative capacity that arises if individual sounds are linked to more than one meaning”

Applying a reality check: this is inconsistent with human language. English and other languages have many ambiguous sounds there / their here / hear one / won two / to / too etc. etc.

Disambiguation through context.

13

Example 1 continued

This model also shows how a limited number of phonemes can be combined to produce an indefinite number of words

“the maximum fitness of a language increases exponentially with word length”

Plotkin and Nowak, J. of Theoretical Biology, 2000, vol 205, p158

Lacks external validity

• Not a model for the evolution of human language

• Could be a model for communication between synthetic agents

14

Example 2 “Distributed Representations, Simple Recurrent Networks, and

Grammatical Structure” Elman, Machine Learning, 1991

Purpose of investigation:“How viable are connectionist models for understanding cognition?” (p.220)

“The connectionist model can be seen as a mechanism for gaining new theoretical insight” (p. 197)

Elman’s model claims to represent long distance dependencies, critical in speech and language.

(e.g. pre-planned co-articulation: lip position for “tea” and “two”)

15

Example 2 (cont)

• Recurrent neural net combines input at time t with previously processed input from time t-n

• Supervised training, using back prop

• Prediction task: What word will come next? Is it grammatical? • Lexicon of 23 words

John feeds dogs.*Boys sees JohnBoys who see John feed dogs.

16

Example 2 (cont)Learning Long Term Dependencies with Gradient Descent is Difficult,

Bengio et al.,IEEE Trans. On Neural Networks, 1994

• It is possible to train a recurrent NN on a particular task

• Models with short dependencies are trainable

• Trade off between efficient learning and latching information for longer periods.

• “gradient descent becomes increasingly inefficient when the temporal span of the dependencies increases” (p. 164)

17

Necessary conditions for scientific simulations

• External validity

• Internal cohesion

• Well founded abstractions

• Fitness for purpose – but what is the purpose?

A simulation might meet these conditions but not be scientific. These conditions are not sufficient.

Consider a simulation of bullying among school boys

18

Bullying scenario (1)

Players adopt the persona of Jimmy Hopkins, a 15-year-old thug who has been incarcerated in a boys’ boarding school.

Points can be scored by terrorising other pupils with a range of physical and psychological abuse. Players use their on-screen persona to kick and punch other pupils and even to spit in their food.

They can use weapons such as baseball bats and catapults. …… The game has angered children’s campaigners.

Sunday Times on the game Bully 14.8.2005

19

Bullying scenario (2)Simulation tool to help victims address bullying problems

Synthetic agents representing school children populate a virtual playground.They develop relationships as they interact - talking, playing and sometimesbullying.

The user identifies with one of the bullied agents and can explore this virtual world, seeing how different reactions might reduce or increase bullying.

Scenario 1 is a game.Scenario 2 may be a scientific search for knowledge about ways of teaching

20

Role of unexpected events in scientific method

Discovery by chance – e.g Galvani and the frogs’ legs, Fleming and penicillin

Recent example:

Behaviour of agents or robots that was unexpected (but can be explained post hoc)

E.g. “Exploiting Physical Constraints: Heap Formation through Behavioural Error in a Group of Robots” Maris and te Boekhorst, IEEE Proc IROS 1996

Experiments with robots on object avoidance failed, but instead delivered results oncreation of clusters. Placement of sensors on robots led to blind spots so that objects were sometimes pushed instead of avoided.

21

ConclusionsScientific method has a bundle of characteristics.

They must include:• A search for knowledge • Objective, repeatable experiments• Logical deductions

They may include • Use of models and simulations that meet necessary conditions of external

validity, internal cohesion, well founded abstractions• Production of falsifiable hypotheses

and should include:• An open mind, ready to expect the unexpected“Chance favours the prepared mind” Louis Pasteur