MODELLING IN AN IMPERFECT WORLD

Luis Willumsen

Planning and policy advice with less-than-rational human beings

CO

NC

ERN

S

Some old concerns

Our track record is not brilliant

J P Morgan found in 14 Case Studies (USA) 2 underestimates (10 to 30% below actual) 4 moderate overestimates (12 to 25% over actual) 8 ‘blue sky’ overestimates (45 to 75% over actual)

• Models are a simplification of reality based on some useful theoretical assumptions and sufficient data to estimate them

• Models are valid insofar the theoretical assumptions remain reasonable; sadly, our theoretical assumptions do not represent human behaviour well

• Therefore, model results are worth little without interpretation and judgement

• So, what do we know about human beings and choices that can help us provide better advice?

CO

NTEN

TS

Contents

1. The key underpinnings of transport demand modelling

2. How travellers really make decisions (the Kahneman model)

Two characters: System 1 and System 2

Two selves: Experiencing and Remembering selves

Two species: Homo Economicus and Homo Sapiens

3. Three contexts for forecasting: Policy Advice

Planning

Forecasting demand and revenue

KEY

REQ

UIR

EMEN

TS

The four pillars of good models

• Good future population synthesis

• System equilibrium

• Consistency of future behaviour

• Behavioural choice modelling

Utility functions and choice models

applied at different levels of aggregation

The parameters in the utility functions

and choice structures remain

the same

Accurate allocation of populations and

activities in the future

Appropriate feed-back through all

relevant submodels to ensure consistent

results

j

m

U

j U

m

eP

e

jq jq jqU V

Forecasting

Tastes and preferences are given and stable, exogenous

to our models

KEY

REQ

UIR

EMEN

TS

The four pillars of good models

From this basis we build a picture of the future that enable us to

compare alternative strategies, projects and policies on a like

with like basis

BEH

AV

IOU

RA

LM

OD

ELS

Utility functions

Modelling choices

The ideal traveller

Is Rational, Selfish and its Tastes do not change, ever.

j

m

U

j U

m

eP

e

j j iU V

A "rational" being that considers opportunities and seeks to optimise

his/her utility by careful choices.

REA

LTR

AV

ELLERS

The real life traveller

A partly rational but also emotional and collaborative being that:

• Cares about changes more than absolute values

• Cannot cope with too many options and uses heuristics

• Has diminishing sensitivity to changes in utility

• Is averse to losses

• Does not react immediately

• But what do we know about this real traveller?

• And how do we adapt our modelling and recommendations to him/her?

REA

LTR

AV

ELLERS

The experiencing self

This is the traveller while travelling

Experience a combination of good and bad aspects of travel

The remembering self

This is what the traveller remembers

Usually salient aspects of the journey

The end of the trips and the results are paramount

Subsequent decisions are more influenced by

what is remembered than the actual journey

itself

TW

OA

SPEC

TSO

FH

UM

AN

DEC

ISION

MA

KIN

GSystem 1 thinking

Intuitive, fast, automatic

Uses heuristics, often answering an easy question rather than a difficult one

Sensitive to changes

Assumes that what you see is all there is WYSIATI

Thoughtful, Logical, requires effort

Lazy, first tendency is to endorse System 1

Can interact with S1 and train it

• iPad and cover cost £550

• iPad costs £500 more than cover

• How much is the Cover?

System 2 thinking

HO

MO

ECO

NO

MIC

US

The Rational Human Being, Homo Economicus

There is strong suspicion that it is a convenient assumption but does not correspond to reality

This mismatch may matter less to develop theory but it does affect the forecasts and advice we provide

• We are not truly Utility Maximisers..

• And we cannot consider all our alternatives..

• We are more affected by changes than by absolute values

• And these changes are based on what we remember from previous experiences, for example delay or price

• Evaluation is relative to a neutral reference point (status quo)

• Diminishing sensitivities to change (Compare £100 to £200 and £1900 to £2000)

• Loss aversion; loses are more onerous than the respective gains

Kahneman-Tversky’s Prospect Theory PR

OSP

ECT

TH

EOR

Y

-16.0

-12.0

-8.0

-4.0

0.0

4.0

8.0

12.0

109876543210-1-2-3-4-5-6-7-8-9-10

Perceivedgain/loss

Generalisedcostchange

Perceivedvaluevscostchange

THERE ARE ALSO OTHER PROBLEMS WITH OUR CURRENT MODELS

ELEC

TRO

NIC

PA

YMEN

T

Fussy prices and money

Santiago tags and gantries

A problem with money...

Separating use from payment crates a different type of money

Ignoring the different kinds of money and prices in our models will lead to wrong forecasts (probably underestimations) and poor advice.

LA

GS

INB

EHA

VIO

UR

AL

RESP

ON

SES

Change job or residenceA problem with time..

Our equilibrium models assume all changes happen at the same time

But people cannot instantly change jobs or homes, and not even time of travel.

We need to recognise the lags in behaviour

But we know and understand little about them

Mode or time of travel

PO

SSIBLE

SOLU

TION

S

Some possible improvements

Hierarchical structure of choices is important for some known biases:

One could give much more important weights to certain attributes in the case of elimination by aspects: Time first, etc..

Nested choices

Choice

PT

Bus

W&R B&B

Metro

W&R B&R P&R

CAR

Owned Club

1 1 1 1 1 1 2 2 2 2 2 2(1 ) (1 ) ..

where and are the parameters for a

loss and a gain respectively, and is equal

1 if the change is a loss and zero otherwise

q

i i

i

V a x a x a x a x

a a

In the case of asymmetric elasticities one can develop a special utility function, even non-linear; but this may create problems for convergence

PO

SSIBLE

SOLU

TION

S

Lags in behavioural change

Hierarchical structure of responses is purpose dependent(?)

For JTW HBShopRoute RouteTime of travel DestinationMode Time of travelDestination ModeFrequency Frequency

Model a time horizon with some of these responses frozen and interpolate

Separate responses

But we know too little about these lags

Cross section data collection is poor at capturing these; this includes SP

We need to learn more from time series and from experimentation

DO

ESIT

MA

TTER?

Does all of this matter?

• Not really, we only want to compare Plans/Schemes/Policies on like-with-like basis that we all agree is good enough

• OK, it is not perfect, but after a little while people do change because of an accumulation of minor disruptions

• There is a trade-off between behavioural accuracy and the equilibrium we need to compare schemes; we vote for consistent comparisons

BUT

o Schemes or plans may affect different responses in different ways

o Sometimes it is important to get the sequence of interventions right

o When forecasting for concessions the right timing and the right response are paramount

IMPROVING INTERPRETATION AND JUDGMENT

IMP

LICA

TION

S

So....

Human nature limits the accuracy of our models

There are implications for Research and for an evolving Best Practice

For Research:

• Develop a better understanding of how uncertainty is affected by the level of disaggregation of our models and data

• Identify lags in behavioural change and develop best ways to deal with them (more social psychology and less mathematics and computational efficiency perhaps?)

• Develop a better relationship between objective (generalised cost) change and perceived loss/gain

• Understand how people switch between System 1 and System 2 modes of thinking in the context of travel

OYSTER

+ GP

SNEW DATA SOURCES WILL HELP

Use of mobile phone, bluetooth, smart card and GPS data

• To monitor performance

• To infer trip matrices

• To study experiments

FO

REC

ASTIN

G

General recommendations on forecasting practice

• Our business is not modelling but forecasting

• We need transport models but our existing tools are less reliable than we pretend; we must acknowledge uncertainty and risk from the outset

• We should start experimenting with the careful adaptation and use of existing techniques to account for more realistic behaviour

• Interpretation and judgement, professional responsibility, should be more open and transparent

• Design and undertake experiments whenever possible, to improve and mediate model results; and this is easier now than in the past

• Document experience more openly

PO

LICY

AD

VIC

E

Policy advice

• Not always depending on modelling

• But our experience should be valuable as it would add analytical rigour to policy discussions

• For example, the issue of Fuel Taxation vs. Road User Charges

• Identification of winners and losers will be more central

• Should experiment with the production of psychological impact evaluation in addition to “objective accounts”

• The role of other “difficulties” of payment, information, familiarity, WYSIATY

• Engage in the discussion of implementation, communication, sequencing and timing (remembering and experiencing self, S1 and S2 thinking modes)

PLA

NN

ING

Transport Planning emerging practice

• Use conventional tools but allow for lags in responses; even with assumed lag rates

• This requires models where certain responses can be switched off at will

• Show and discuss the impact of these lags and if critical look for other approaches to settle the choice of plan/scheme

• Identify winners and losers and by how much

• Account separately for large and small loses/gains

• Acknowledge uncertainty and the risk of over-calibration and spurious precision

FO

REC

ASTIN

G

Forecasting Traffic and Revenue

• Our track record is better than that of bankers and regulators

• But it is still not that good

• Acknowledge uncertainty and risk from the outset: identify sources of risk, estimate their importance and focus on reducing them

• Disaggregate for willingness to pay but do not over-complicate the model

• Careful use of existing techniques, even with the limitations shown, is a reasonable approach. But, support forecasts from different complementary perspectives

• For example, a classic model forecast, a trend extrapolation forecast and benchmarking against similar systems

• Undertake risk analysis

RISK

AN

DD

EMA

ND

CO

NTR

IBU

TOR

S

De-construct contributors to traffic: conceptual LRT forecasts

STO

CH

ASTIC

RISK

AN

ALYSIS

Stochastic risk analysis

Beloved by Financial Institutions

Generally based around Base Case

Take 2-3 key variables : GDP, SVT, etc

..and look into their historical variability (standard deviation )

The model is used to track variability in revenue resulting from variability in key inputs, usually via a simplification in Excel

FutureRevenue = RevenueFactor * Base Case Revenue

A value for RevenueFactor of 1 indicates Base Case

Also presented as the level of revenue that is likely to be exceeded 90 or 95% of the time (P90 and P95)

TYP

ICA

LO

UTP

UT

Example of output

0.85

0.9

0.95

1

1.05

1.1

1.15

2003 2005 2007 2009 2011 2013 2015

Year

Re

ve

nu

e F

ac

tor

GDP variation σ: 0.5%

SVT variation σ : 2.5%*mean VST

FO

REC

ASTS

CA

SE2

P90 and P95 for toll road

'Four Roads- Case 2 WRe P90 sd 1.5

2

7

12

17

22

2008 2010 2012 2014 2016 2018 2020 2022 2024 2026 2028 2030 2032 2034 2036

Mx$ B

illio

ns

RO

UN

DU

P

Round up

1. Some of these risk analysis techniques will also filter through into normal transport planning models and practice Especially for key projects like HS2

2. Fundamental research into real travel behaviour and choices is necessary

3. Improvements to current practice that recognises some limitations of our models are possible and desirable

4. Benchmarking and well documented experience elsewhere will be used more often to support forecasts

5. This will be facilitated by new data sources and electronic trails

6. Modellers should engage more with real issues and develop reliable judgement and interpretation skills; this may require adaptation of training programmes

THANK YOU