towards integrating everything (well at least: abm, data-mining, qual&quant data, networks and...

Towards Integrating Everything, Bruce Edmonds, Joining Complexity Science And Social Simulation For Policy, Budapest, May 2014. slide 1

Towards Integrating Everything…

Bruce EdmondsCentre for Policy Modelling

Manchester Metropolitan University


…well at least: ABM, data-mining, qualitative & quantitative data, networks and complexity

science


Complexity Science and Policy Do Not Mix Well..

Because (among other reasons):• The phenomena of interest to policy makers is

complicated, complex and changing all the time and thus very hard to “understand”

• Complexity Scientists and Policy People have very different goals (understanding vs. action)

• They have to work in very different ways with respect to very different sub-cultures/peers

• They are (mostly) unwilling to understand each others’ world

• There is a tussle between them in terms of status and power (control)


Some example problems

• “You have 3 months to give me your best forecast, however preliminary”

• Policy makers are unwilling to outsource any control over the process to researchers

• Important data is not available to researchers• Policy makers already know what they will do, they

are just looking for a justification or story• Complex models make policy debate difficult• The outsourcing of blame: “The decision was made

based on the best scientific advice”• The worry of researchers that the caveats underlying

their models will be lost/ignored


This talk is about some different strategies for dealing with this

• That is, to step back, and look at what we are doing in complexity science and how this might (or might not) ‘fit’ into the world of policy making and hence usefully inform policy

• In other words, can our models and understanding be useful to society and, if so, how

• Now, unfortunately we have only a little “data” about this relationship – there has been relatively little observational work on this

• Thus I will proceed by looking at some of the issues and discussing possibilities only

• This is a synthetic talk, amplifying some trends and issues already raised at this workshop then extrapolated to a conclusion (that not all will be happy with)


Strategy 0: A natural language discourse

• Issue is discussed in rich, meaningful language• Any models are implicit and/or analogical• Meaning of terms is often vague or ambiguous

– Advantages in terms of building consensus– Disadvantages when working out what went wrong

• Debate is accessible to everyone• But lack of cumulative knowledge development• Depends upon “conceptual framework” and hence can be

influenced by fashion• Not much good for anticipating radical change• Good for integrating diverse evidence albeit informally• But poor at dealing with complication/complex detail


Strategy 1: “predictive” “black-box” macro-level models

• Model the system relating macro-level properties of the whole system (using differential equations, rate equations, systems dynamics etc.)

• With a view to predicting the effects of different policy options (albeit with large error bounds)

• It is possible to make such models, of social phenomena but understanding is often not the main way of doing this but trial&error – in other words model adaption (Nate Silver)

• But this only works if nothing essential changes – these models only give “surprise free” projections


Strategy 2: partial, complex, micro-level models used as an analogy

• Model some processes/aspects at the micro-level system to observe the emergence of outcomes

• Does not easily relate to data, does not predict, and remains quite abstract from the observed

• Can be used to understand/explore the possible model “trajectories” of outcomes

• But the model remains more of an analogy, because the mapping to any observed case is unclear and remade by each interpreter

• This gives a “story” why, but is difficult to relate to particular policy options/questions

• Tends to give “negative” conclusions w.r.t. policy


Kinds of Laws

• “How the Laws of Physics Lie” N. Cartwright, 1980• Strategy 1 corresponds to “Phenomenological

Laws”, they match/predict the observed data but do not explain

• Strategy 2 corresponds to “Explanatory Laws”, they explain why things occur but do not predict

• You need both kinds of model but crucially also the “bridging rules” learnt via acculturation connect the two – this is often not explicit

• In a mature science formal derivations/theories can be made between the two kinds of model


An illustration: Modelling a gas

T

V

The gas laws:

A Phenomenological Model

Picture of randomly moving, weakly interacting elastic spheres

An Explanatory Model

Bridging rules


Back to Complex Social Science

• People are applying these two types separately, indeed they are often in different academic fields

• The bridging rules have not been developed• Broadly the policy world wants the properties of

strategy 1 models• …whilst complexity science likes strategy 2• This reflects the difference in their goals and both of

their wishes to retain control and avoid blame• But neither, on their own, will satisfy the joint needs of

complexity and policy worlds, they are either: unable to deal with structural change or they only provide a kind of analogy as to what might happen


Robots in Uncertain Environments

• AI/robotics in the 60s/70s applied an approach where the entity kept a model of its world and tried to evaluate different alternative actions via predicting their effects using the model

• However these were not good at coping with unknown and uncertain environments

• Rather what turned out to work much better was:– Using the world as a model of itself – frequent sampling– Fast adaption in response to immediate conditions– Different levels of abstraction and/or control

(subsumption architecture)


Strategy 3: “consistent” views of what is happening

• Not general “theories” of social phenomena• Rather well-understood and “consistent”

abstraction from the data that gives a “view” that is designed for a particular purpose

• Behind this there will be a (maybe implicit) model which explains how the resulting view is generated from the source data

• But this model does not have to capture the processes within the observed phenomenon

• And, crucially, it is not a projection forward in time but a view of the current situation


This can involve different kinds of data manipulation

Including:• abstraction:

– losing information to make patterns/trends more visible– selecting aspects relevant to the present purpose

• synthesis: integrating data from different sources to produce:– a more complete picture– a more meaningful picture

• re-expression: manipulating the data to:– suit the interpretative abilities of humans– show how key aspects/properties relate/change


Desiderata for such “views”

• Timely: to be able to be produced reasonably quickly (once designed!), so it is practical to use

• Adequate: to give enough information about the current state and (maybe) how it is changing

• Consistent: in the sense that the same kind of situation is reliably recognisable from its presentation (given its purpose)

• Transparent: The meaning of what they show needs to be clear enough to be learnable

• Well-founded: in the sense that it is clear how to adapt it to new/changed data


Use of these “views”

• The algorithm for constructing these views might well be derived from the results of extensive analysis, simulation modelling etc.

• However, they are themselves a more straight-forward representation of the data with a clear and comprehensible account of their nature

• They are used to help “steer” policy by giving a richer but timely sense of what is happening

• Policy makers (maybe initially with advice from researchers) learn how to interpret the views

• They “own” the view machinery and its use, not the researchers

• The views might be publically available, so that their import may be discussed


An illustration of Strategy 3 in action

Modelling micro-

aspects

Data analysis

Expert opinion

ABM and other

analysis

Understanding processual possibilities

Measures, tools and

visualisations

Policy Decisions

Consequences

Wider Public

Policy WorldResearch World


In order to realise this strategy…

…we need to:• integrate the streams of evidence/data• discover what the possibilities in terms of social

processes are (using data mining, simulation modelling etc.)

• use this to focus on what would indicate the early onset of these processes occurring and their progress

• present tools for these using a variety of means (visualisations, statistics, graphs, interactives etc.)

• but in a targeted and relevant manner• then be willing to let go an others develop their use

ABM and other techniques

What is delivered to the policy world


Example: Understanding Innovation

Innovation is a multi-level phenomena, including:• Individuals creating new things/ideas/innovations• The spread of these over networks• How innovations are received – how they ‘fit’ into the

conceptual and pragmatic frames of the potential recipients

• How products/ideas combine to deliver services• How some innovations are catalytic/tools – facilitate the

creation of many more innovations• How innovations can change the affordances and

habits of a whole society and hence influence its culture• etc. etc.


Narrative Data

• Observational or personal accounts of what people do and their motivations are valuable source material, especially for the specification of the micro-level (agent behaviour)

• There is a wealth of such evidence within qualitative research, albeit often “smothered” within (what seem to us as outsiders) obscure jargon, issues, debates and pretentions

• This can be used to suggest some of the menu of strategies people use and when they use them

• Are often relatively mundane and context-specific• They might be mistaken, but are often a much better starting point

than the theories of academics• Methods for using such narrative data to inform the specification of

agents are currently being worked upon• These can, to some extent, bring some of the “mess” of observed

social life into ABM models


Psychological Evidence

• Often the constructs we use to make our agents are ad hoc, coming from a “folk psychology”

• They are not checked for consistency against what is known in psychology

• Unfortunately (for us) the psychological literature is concerned with its own issues and debates and does not often give the answers we need

• And it tends to look at different aspects in isolation and gives no clue as to how they relate/integrate

• However it does provide weak constraints as to the specification of agent behaviour and a language within which to critique suggestions


big Data

• can provide fine-grained but noisy and incomplete information about actions

• data-mining these can provide insights about how people behave

• but provide richer insights when analysed with, or compared against, other evidence and hypotheses

• in particular, clustering of such data might indicate some of the different kinds of behaviour

• each of which might be then abstracted to suggest some different behavioural rules


Survey Data

• Existing surveys are often not very informative or useful, only providing veiled hints for what people might be doing/thinking

• The detail data from the survey is more useful than the particular summaries presented in paper

• However, once the relevant behavioural dimensions and strategies are known directed surveys can be useful to determine proportion of occurrence and conditions of occurrence


Network Data

• Direct network data is only sparsely available and then often very incomplete

• Or derived from big Data where the connection between links suggested by that data and social connections in the target phenomena is weak

• But can be a useful validation check on the networks that emerge from ABMs

• The relevant aspects of these networks could be compared with those in available data

• However, comparing networks is non-trivial• Even when sampled in odd ways, the sampling can

be mimicked within the ABM then compared


Participatory Validation

• When results are presented/developed with stakeholders and/or subjects

• Can provide valuable input as to the meso-level output/behaviours

• Context is important to provided –histories and situations wherein the behaviour might occur so that participants can give meaningful responses

• Often useful in getting indications of where a model might be wrong

• Animations/visualisations/graphs are important to give a fuller picture to participants


Time Series Data

• Are usually at an aggregate or group level• Gives an idea of the dynamics one is looking for• Are often quite derived/abstract

Aggregate Statistics

• Aggregate summaries of measures• Give different projections of the data• Are most useful when most closely linked to policy

inputs or (un)desirable outcomes


Integration via ABMs to “Views”

Micro-levelNarrative data

Psychology

Data-mining

Survey data

Network data

Participatory input

Meso-level

Macro-level

Time-series

Aggregate Statistics

Survey summaries

ABM etc.

Archetypal Stories of Individuals

Complex Visualisations

Key Global Indicators

Del

iver

ed t

o P

olic

y W

orl

d

Scenarios


Side Note: Social Context

• An obsession of mine• Word “Context” is problematic, but roughly the

perceived kind of situation (e.g. keynote lecture)• Context is not necessarily accessible to conscious

thought, reifiable, consistent or abstractable• But some are socially entrenched and obvious• People behave differently in different contexts!• Identifying social contexts (via data mining,

observation etc.) is key to integrating many data, particularly qualitative or specific data with more generic/abstract patterns


Being A Modeller

• Status is largely accorded to higher end abstractions rather than the concrete/mundane

A more balanced science

Social Simulation/Complex

ity Science

Abstract constructions

Concrete constructions

• We like to maintain control over our own constructions including their subsequent use

• We inevitably see the world through our models and so distort as we make progress


Some things I am suggesting to avoid

• Giving policy makers/advisers predictions is as unwise as giving a sharp knife to a child – at best it will not be useful to them and at worst it could cause a horrible accident and it will be your fault

• Just because a model provides a good way of thinking about things does not make it true.

• Policy makers expect scientists to only present only (basically) correct models to them – not unvalidated speculations or computational analogies (ways of thinking about something).

• Even though you find a model really interesting and revealing, do not attempt to share the model with those involved in policy – they have fundamentally different concerns and goals to you. This is not just a matter of expressing things clearly/simply enough.


Conclusions

• The product should not be a model so much as a representation of what is happening

• That is clear and the result of informed research• But then ceded into the policy

development/assessment cycle, losing control over it

• Thus to avoid relieving policy makers of the burden decision making and future uncertainty

• In particular not to predict, even with error bars• but to inform them better as to what is happening• this will involve us shifting to data integration,

analysis and presentation more than modelling


The End!

Bruce Edmonds:http://bruce.edmonds.name

Centre for Policy Modelling:http://cfpm.org

The slides will be/are uploaded at: http://slideshare.net/BruceEdmonds

towards integrating everything (well at least: abm, data-mining, qual&quant data, networks and...

Education