Four different views of a policy modelan analysis and some suggestions

Bruce EdmondsCentre for Policy Modelling

Manchester Metropolitan University

Two Worlds

Four different views of a policy model: an analysis and some suggestions, Bruce Edmonds, Policy Workshop@ECCS, Vienna, September 2011, slide 2

Research• Ultimate Goal is

Agreement with Observed (Truth)

• Modeller also has an idea of what the model is and how it works

Policy• Ultimate Goal is in

Final Outcomes (Usefulness)

• Decisions justified by a communicable causal story

Policy Model• Labels/Documentation may be

different from all of the above!

Modeller Policy Advisor

Four Meanings (of the PM)

Research World

1. The researcher’s idea/intention for the PM

2. The fit of the PM with the evidence/data

The ideavalidation relation extensively discussed within research world

Policy World

3. The usefulness of the PM for decisions

4. The communicable story of the PM

The goalinterpretation relation extensively discussed within policy world

Four different views of a policy model: an analysis and some suggestions, Bruce Edmonds, Policy Workshop@ECCS, Vienna, September 2011, slide 3

Exemplars of the Divide between Researchers and Policy Makers/Advisors

• Policy makers try to get researchers to predict the unpredictable

• Researchers can’t get the input (time, money, data) they need from policy makers

• Policy advisors want support for a policy (their best guess that it’s good) and researchers reply with caveats and complications

• Policy Makers (over) simplify the results• Researchers make the model so complex

nobody but they can understand itFour different views of a policy model: an analysis and some suggestions, Bruce Edmonds, Policy Workshop@ECCS, Vienna, September 2011, slide 4

Two Models of Truth

Realism• Truth through relationship

of output to observations• Feedback through

correspondence (RMSE error, stylized facts, knowledge about processes)

• ForecastObserve loop• Often associated with hard

sciences, numbers and analytic models

Instrumentalism• Truth through trying

something and assessing how well it works

• Feedback through success (cost, votes, pleasure, pain)

• ActionAssess loop• Often associated with the

humanities, narratives, politics, perspective and aggregate statistics

Four different views of a policy model: an analysis and some suggestions, Bruce Edmonds, Policy Workshop@ECCS, Vienna, September 2011, slide 5

The Anti-Anthropocentric Assumption

• That the universe is not arranged for our benefit (as academics trying to understand it)

• That assumptions as these are likely to be wrong:– Our planet is the centre of the universe– Planetary orbits are circles– Risky events follow a normal distribution

• But here… that the social phenomena we study happen to be such that numerical and other easy ‘surface’ methods of understanding will be sufficient

• In other words, we can not side-step the difficulties with some clever proxies/mathematics/statistics/etc.

• Simplicity (what is easier) is not any guide to truth

Four different views of a policy model: an analysis and some suggestions, Bruce Edmonds, Policy Workshop@ECCS, Vienna, September 2011, slide 6

Consequences of the AAA

• There is no reason to suppose that a model that is adequate to modelling policy issues will be simple enough for us to understand– (e.g. that it will be analytically solvable)

We may well have to abandon hope of generic models and settle for context-specific approaches

We will need to use a combination of different approaches

We will need to model our models etc. to have a change of understanding them

Four different views of a policy model: an analysis and some suggestions, Bruce Edmonds, Policy Workshop@ECCS, Vienna, September 2011, slide 7

Some modelling tensions

precision (model not vague)

generalityof scope

Lack of error (accuracy of results)

realism(reflects knowledge of processes)

Economic Models

Scenarios

Agent-basedmodels

Stats/regressionmodels

Reality?

Wanted forpolicy decisions

Four different views of a policy model: an analysis and some suggestions, Bruce Edmonds, Policy Workshop@ECCS, Vienna, September 2011, slide 8

But this is the same as what confronts any organism/robot…• An adaptive entity faces an environment

which is overwhelmingly complex, uncertain, unknown etc.

• Thus this is a dilemma that AI/Robotics also confronts… whether to make robots:– Intelligent, learning knowledge about its world

and reasoning (i.e. a realist approach)– Or “hard-wired” with some imperfect but

effective tricks to enable it to survive and do the essential (i.e. an instrumental approach)

Four different views of a policy model: an analysis and some suggestions, Bruce Edmonds, Policy Workshop@ECCS, Vienna, September 2011, slide 9

AI/ML/Robotics/Adaptive Behaviour etc.

• They are deeply split about the best approach, with competing traditions that do communicate

• Started with reasoning and problem solving, assuming correct knowledge was at hand (a strictly realist limited-context approach)…

• …but in more challenging domains, quick and dirty instrumentalist approaches (following Brooks) seemed to do better but these had limitations and were specific to a certain goal

• They have not resolved this issue…• …but at least the competing approaches are recognised

and much discussed.• So here are some outcomes from that discussion…

Four different views of a policy model: an analysis and some suggestions, Bruce Edmonds, Policy Workshop@ECCS, Vienna, September 2011, slide 10

1: the Symbol Grounding Problem

• Harnad (1990) Physica D• Formal representation can not gain meaning from

internal relations with other part of the formal representation

• In order to attach internal symbols to their meaning one needs a repeated and frequent interaction in context– By repeatedly changing a model with respect to the

available evidence – By repeatedly using the model for decision making and

seeing the results– Via frequent participatory methods, involving policy

makers/advisors/stakeholders in the modellingFour different views of a policy model: an analysis and some suggestions, Bruce Edmonds, Policy Workshop@ECCS, Vienna, September 2011, slide 11

2: Starting “small”

• Elman (1993) Cognition• Start with easy and specific situations and

learn about them• Does not matter if these are instrumental

and largely wrong, but the incremental building up to more useful domains essential

• Only generalising to harder and more general situations slowly

• In other words, a commitment to a continual and lengthy modelling process

Four different views of a policy model: an analysis and some suggestions, Bruce Edmonds, Policy Workshop@ECCS, Vienna, September 2011, slide 12

3: Multi-Level Modelling

• Brooks (1991) Artificial Intelligence• Has fast local and specific reactive circuits for

control of short-term behaviour• With layers that coordinate these lower level

control strategies at progressively higher levels• Lower levels continue what they are doing

feeding back to higher levels• Higher levels occasionally adjust or interrupt

lower level units• Resulted in robots that could navigate rough and

unknown domains like insectsFour different views of a policy model: an analysis and some suggestions, Bruce Edmonds, Policy Workshop@ECCS, Vienna, September 2011, slide 13

A Layered approach to modelling

The phenomena of

concern

The phenomena of

concern

SimulationSimulationSimulation

Social Network Models

Social Network Measures

Four different views of a policy model: an analysis and some suggestions, Bruce Edmonds, Policy Workshop@ECCS, Vienna, September 2011, slide 14

A tighter loop involving stakeholders...• Stakeholders are involved in parts of the

“modelling” loop: criticising model, providing data, specifying model, determining goal etc.

• Involvement comes from: relevance to their goals, having some effect/control, quickly seeing the results, feeling involved, not onerous, being situated in their lives

• This inevitably means a loss of control by modellers! This is unavoidably political.

• A radical move: giving this power more directly to people rather than their representatives

Four different views of a policy model: an analysis and some suggestions, Bruce Edmonds, Policy Workshop@ECCS, Vienna, September 2011, slide 15

Two Worlds –

Empirical• Ultimate Goal is

Agreement with Observed (Truth)

• Modeller also has an idea of what the model is and how it works

Instrumental• Ultimate Goal is in

Final Outcomes (Usefulness)

• Decisions justified by a communicable causal story

Model• Labels/Documentation may be

different from all of the above!

Tighter loop =

participatory modelling

Four different views of a policy model: an analysis and some suggestions, Bruce Edmonds, Policy Workshop@ECCS, Vienna, September 2011, slide 16

Top-level Policy Advisors, Academics, Official Stats.

Crowd sourcing/input, individuals

Regional-level Policy Makers, Stakeholders

A Multi-Level Vision of PM

Top-level PM

Topic Specific PM

Region Specific PM

Citizen-level PM

Citizen-level PM

Citizen-level PM

Citizen-level PM

Top-level Policy Makers

Local Stakeholders, activists

Feedback on Use

ContinualDocking

ContinualDocking

ContinualDocking

Empirical Input

Pressure groups, Qual. Data, Academics

Four different views of a policy model: an analysis and some suggestions, Bruce Edmonds, Policy Workshop@ECCS, Vienna, September 2011, slide 17

Consequences of Multi-Level PM

• Different models being “pulled” in different ways by different groups, inputs and needs

• Continual re-modelling to keep models ‘docked’ with each other and to incorporate new observed processes (maybe with a distributed ‘wiki’-like structure)

• A lot of work by stakeholders as well as researchers

• A lot of data of ALL levels and kinds: textual, anecdotal, network, aggregate statistical, mass data, time-series etc.

Four different views of a policy model: an analysis and some suggestions, Bruce Edmonds, Policy Workshop@ECCS, Vienna, September 2011, slide 18

Main Conclusions

• The Policy/Research divide is deep, not just a matter of inclination, stubbornness or training, but rooted in their different goals

• Lessons from the AI/Robotics world imply:– Need for tight and sustained iteration of

modelling processes– Start with easier problems, work up later– Multi-level approach, with different levels of

model, generalising only when possible and necessary

Four different views of a policy model: an analysis and some suggestions, Bruce Edmonds, Policy Workshop@ECCS, Vienna, September 2011, slide 19

Auxiliary Conclusions

• A need for techniques (human and computer aided) for “translating” qualitative evidence from conversations and texts to the agent rules in simulations

• The need for an ethnographic observation and study of researchers and policy people interacting

• Possibly writing “manuals” for policy advisors “Researchers: how to deal with them, their advantages and bugs”

• A similar manual for researchers about policy advisors/makers!

Four different views of a policy model: an analysis and some suggestions, Bruce Edmonds, Policy Workshop@ECCS, Vienna, September 2011, slide 20

The End

Bruce Edmonds

http://bruce.edmonds.name

Centre for Policy Modelling

http://cfpm.org

Slides uploaded to http://slideshare.com