why do models fail? problems, problems …

Kingsley Dunham CentreKeyworthNottingham NG12 5GGTel 0115 936 3100

© NERC All rights reserved

Why do models fail?Problems, problems …

John ReesBGS Head of Policy and Science Co-ordination

Andrew HughesBGS Groundwater Modeller

and NMPI Co-organiser


Ideal vs reality


Generic issues as seen from a government

institute - the British Geological Survey


BGS and numerical modelling

Groundwater - Model Developer and user Coastal change - Poacher and gamekeeperContamination - often the ‘honest broker’


Typical reasons for failure

• Lack of consistency of approach

• Turnover and education of staff in client organisations

• Language between user and modellers, and between modellers

• Appropriate and sufficient data

• Trust between parties

• Unrealistic expectations

• Honesty about the limitations of models

• Cost of taking-up new developments


\Policy makers

\Resource Managers

\Modellers


Key organisations

• Universities

• Consultants

• Government Institutes

• Regulators

• Utilities

• Government Ministries

• Pan-governmental


Regulator issues

• Limited number of staff that have modelling skills

• Accessibility of models is often poor – they are often not easy to run

• The maintenance and updating of models has a high cost (the leaky roof syndrome)

• Inadequate data to support development or understanding

• Expectations are often not met

• Stakeholders are not adequately consulted


Resource manager issues

• Common uncertainty about the specification and scale of model required

• Lack of clarity about the needs of regulators

• Find that models have too much uncertainty for detailed use

• Inadequate timely stakeholder involvement

• Find that models are not as flexible as managers would like


Model developer issues

• Models are driven by policy and not the other way round e.g. Habitats Directive

• Few examples of models being trusted by all parties

• Stakeholders are recognised as important, but involvement is very variable

• Personalities are important in defining how much stakeholders interact

• Managing expectations is very important

• Those consultants who deliver to spec and on time may not be the best to drive modelling forwards


Broader implications

• Lack of efficiency associated with development encourages adoption or tweaking of older ‘industry standard’ models instead of adoption of newer models.

• Drive to conservatism encourages usage of ‘tried-and-tested’ consultants who focus on delivery, rather than more innovative scientists who will introduce new concepts and stretching the modelling.

• Acceptance of limitations (e.g. empirical constants, black-boxes) instead of driving better modelling.


Towards solutions• Need common understanding before attempting to

formulate solutions

- Climate change models are not taken up by some national governments that do not accept reality!

• Language – differs markedly between disciplines

• Definition of problem type, organisations involved and geographic extent

• Problem is potentially huge, so need boundaries

• Guidelines are needed


Summary


Questions ?

why do models fail? problems, problems …

Documents