assessment of transport projects: risk analysis … kim salling.pdfassessment of transport projects:...
TRANSCRIPT
Assessment of Transport Projects: Risk Analysis and Decision Support
Assistant Professor
Kim Bang Salling
Presentation at the RISK Conference 2011:
December 1st 2011
DGI Byen, Copenhagen
DTU Transport, Kim Bang Salling 2
Outline
• Background
– Introduction
• Methodologies
–Cost Benefit Analysis (CBA)
–Quantitative Risk Analysis (QRA)
• Feasibility Risk Assessment (FRA)
–Accumulated Descending Graphs (ADG)
• The UNITE-DSS Decision Support Model
–Uncertainties in Transport Project Evaluation
• Conclusion
• Perspective
DTU Transport, Kim Bang Salling 3
Background
• The Manual for socio-economic analysis in the transport sector (2003)
–Unique guidelines for evaluating transport infrastructure projects
–Lack of uncertainty handling
–Expected revision 2012-2013
• Building decision support ”with a twist”
–Rational decision making involves the assessment of both the benefits and the losses (costs)
–The need for making ”good” decisions in transport planning and evaluation are vital
DTU Transport, Kim Bang Salling 4
Transport Planning and Assessment
Decision
support
Ongoing
transport
planning:
- Societal goals
as, for example
networks and
mobility,
sustainable
development,
etc.
- Prognoses/
forecasts
- Urban &
regional
planning
- Design
standards, etc.
Transport
infrastructure
project
proposal
Traffic
models
Impact
models
Multi-criteria
analysis
(MCA)
Cost-benefit
analysis
(CBA)
Research:
- Concepts as
for example
Feasibility Risk
Assessment
(FRA) and
Accumulated
Descending
Graphs (ADG)
- The CBA-DK
model and
@RISK software
- Case examples
related to
different modes
- Findings and
recom-
mendations
DTU Transport, Kim Bang Salling 5
Introduction
• CBA & MCA produce single point estimates
• Informativ decision support
– Feasibility Risk Assessment (FRA)
– Accumulated Descending Graphs (ADG)
• Normally, uncertainties are handled by sensitivity tests
• Historical overview of uncertainties
– Construction cost ’overrun’
– Traffic forecast ’underrun’ (traffic modelling)
DTU Transport, Kim Bang Salling 6
Construction Cost Overruns (fixed prices)
Construction Cost Overruns
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Channel Tunnel,
UK & France
Øresund Access
links, DK &
Sweden
Great belt link, DK Øresund coast-to-
coast link, DK &
Sweden
Co
st
Ov
err
un
(%
)
Construction cost overruns
0%
200%
400%
600%
800%
1000%
1200%
1400%
1600%
1800%
2000%
Su
ez C
an
al
Syd
ne
y
Op
era
Ho
use
Co
nco
rde
Su
pe
rso
nic
Ae
rop
lan
e
Bo
sto
n's
Art
ery
/Tu
nn
el
Pro
ject,
US
A
Hu
mb
er
Bri
dg
e,
UK
Bo
sto
n-
Wa
sh
ing
ton
-
Ne
w Y
ork
Gre
at
Be
lt
Ra
il T
un
ne
l,
DK
A6
Mo
torw
ay
Ch
ap
el-
en
-le
-
Fri
th/W
ha
ley
Sh
inka
nse
n
Jo
ets
u R
ail
line
, Ja
pa
n
Wa
sh
ing
ton
me
tro
, U
SA
Ch
an
ne
l
Tu
nn
el, U
K &
Fra
nce
Ka
rlsru
he
-
Bre
tte
n lig
ht
rail,
Ge
rma
ny
Øre
su
nd
Acce
ss lin
ks,
DK
& S
we
de
n
Me
xic
o c
ity
me
tro
lin
e,
Me
xic
o
Pa
ris-A
ub
er-
Na
nte
rre
ra
il
line
, F
ran
ce
Co
st
Ov
err
un
s (
%)
DTU Transport, Kim Bang Salling 7
Cost-Benefit Analysis (CBA)
• Method for evaluating the ”goodness” of investments
– A systematic approach in listing costs and benefits
– Selection of the ”best” performing alternative(s)
• Inputs derived from a lot of external sources
– Traffic models and impact models
– Key figue catalogues
• Output based upon single point criteria
– Net present value (NPV)
– Benefit cost ratio (BCR)
• Transferred model uncertainties!?!?
DTU Transport, Kim Bang Salling 8
Uncertainty in transport appraisal
• Unit price principles are assumed ”certain”
• Two types of impacts stands out: – Travel time savings -> Benefit
– Construction costs -> Cost
• Literature supports the latter
impacts by the so-called:
Optimism Bias
Sources of Uncertainty
Unit Pricing
PrinciplesModel Uncertainty
Relies on the key figure
catalogue in calibrating
and determining unit
price settings.
Relies on the model
build up of impact and
traffic models that
provide the input
towards decision
support models.
Randomness of the
systemLack of knowledge
DTU Transport, Kim Bang Salling 9
Optimism Bias and Reference Class Forecasting
The Transport Planning Phase: Adapted from the British Department for Transport (DfT) (2004)
Reference Class Forecasting: Optimism Bias
Inside View Outside View
”Uniqueness” of Project
”The Planning Fallacy”
Reference Class
Forecasting
Forecasting of particular
projects
Forecasting from a group
of projects
(1) Identification of
relevant reference
classes
(2) Establishing
probability
distribution
(3) Placing and
comparing the
project
Optimism Bias UpliftsCurrent Situation
DTU Transport, Kim Bang Salling 10
Optimism Bias and uplifts
• Deriving uplifts is highly dependet on large data-sets
–Flyvbjerg from (AAU) has since 2003 developed a large database
–Unfortunately, it looks upon mega-projects
• Uplift values were derived on basis of Reference Class Forecasting i.e. statistical measurements on various project pools
• Applying uplifts still produces single point rate of returns
• BUT, the data collected can be transformed and used in another way….
Risk Analysis
DTU Transport, Kim Bang Salling 11
Risk ”Control” – Infrastructure assessment
General information
(Technical, political,
economical, etc.)
”SAFETY” SOCIETAL GOALS:
PHILOSOPHY:Definition of goals,
fundaments for priorities
and standards
ACCEPTANCE
CRITERIA:Societal acceptance,
budgetary constraints etc.
Appraising the
information brought
above
RISK ANALYSIS (TRADITIONAL):
RISK
IDENTIFICATION:Definition of risk
components - impacts
RISK
ASSESSMENT:Describe and quantify risk
by evaluation
RISK
EVALUATION:Compare risk to
acceptable standards
TRANSPORT INFRASTRUCTURE PROJECT:
DTU Transport, Kim Bang Salling 12
Monte Carlo Simulation
DTU Transport, Kim Bang Salling 13
Input Distributions
• Distinction between non-parametric and parametric
–Non-parametric is used when experts have to make the judgments
–Parametric are used when data and/or theory underpins the judgments
• Non-Parametric distributions:
–Uniform
–Triangular/Trigen
• Parametric distributions:
–Normal
–Erlang (Gamma) –> Construction Cost
–PERT (Beta) -> Travel time savings
DTU Transport, Kim Bang Salling 14
Level of Knowledge (LoK)
• The LoK ranges from low to medium to high
• Distinction between Parametric and Non-Parametric distributions
DTU Transport, Kim Bang Salling 15
PERT Distribution
• Based upon a beta distribution with the assumption that the mean can be derived from:
• This makes it ideal for modelling experts opinion
–Stands out compared to the Triangular distribution
6
4 MaxModeMinMeanPERT
Triangular
Beta-PERT
3
MaxModeMinMeanTriang
vs
DTU Transport, Kim Bang Salling 16
Data fit (Rail) – Demand forecasts
• Demand forecasts (user benefits) are set against prior Reference classes derived from Flyvbjerg et al. (2003)
• 27 rail projects were compared where the inaccuracy on average were 39% lower than predicted
• I have fitted a PERT curve around the data from Flyvbjerg et al. (2003)
DTU Transport, Kim Bang Salling 17
Data fit (Road) – Demand forecasts
• 183 road projects were compared where the inaccuracy on average were 9% lower than predicted
Fit Comparison for Inaccuracy in Traffic ForecastsRiskPERT(-78.5;9.6%;179.34%)
1,057-0,4875,0%90,0%5,0%
-150%
-125%
-100%
-75%
-50%
-25%
0%
25%
50%
75%
100%
125%
150%
175%
200%
Input Beta-PERT
DTU Transport, Kim Bang Salling 18
Erlang Distribution
• Based upon a gamma distribution defined upon a shape and a scale parameter (k, )
• The shape parameter, k, depicts the skewness of the distribution whereas the scale, , is based upon data
0
0.5
1
1.5
2
0 0.5 1 1.5 2 2.5 3
K=2
K=5
K=10
K=20
DTU Transport, Kim Bang Salling 19
Data fit (Rail) – Investment costs
• Flyvbjerg et al. Compared 58 rail projects
• Approximately 88% of the probability mass is above 0 which indicates that rail type projects are underestimated
• The fitted probability distribution contributes to the fact that an Erlang distribution is very well suited
DTU Transport, Kim Bang Salling 20
Data fit (Road) – Investment costs
• 167 road projects were compared where the inaccuracy on average were 20% lower than predicted, with k = 8
Fit Comparison for Cost Overrun for Road ProjectsRiskErlang(8;0.09) -> (-33.6%;20.2%;222.6%)
0,569-0,1565,0%90,0%5,0%
-100%
-75%
-50%
-25%
0%
25%
50%
75%
100%
125%
150%
175%
200%
225%
250%
Input Erlang
DTU Transport, Kim Bang Salling 21
Recommendation – High level of knowledge
• Risk analysis in decision support:
– Combination of data from Flyvbjerg, Successive Principle and Risk Analysis: Large-scale implementation in UNITE
– Definition of distributions
– Empirical data to feed the distributions
• Assigning probability distributions:
– Investment Cost – Gamma (Erlang) distribution
– Travel Time Savings – Beta (PERT) distribution
Mode Impact Distribution Low High
Rail Travel time savings PERT -90% 140%
Rail Construction cost Erlang (k = 23) -40% 120%
Road Travel time savings PERT -80% 180%
Road Construction cost Erlang (k = 8) -30% 120%
A negative sign for travel time savings means that benefits have been overestimated and a negative sign for construction costs
means that costs have been underestimated
DTU Transport, Kim Bang Salling
Uncertainties in Transport Project Evaluation (UNITE)
22
Uncertainties in Transport Project Evaluation (UNITE): the five Work-Packages
(5) Evaluation methodology
WP5 project leader: Steen Leleur (DMG)
(4) Uncertainty calculation in transport models
WP4 project leader: Otto Anker Nielsen (TMG)
(2) Organizational context of Modelling, an
empirical study
WP2 project leader: Petter Næss (AAU)
(3) Uncertainty calculation of cost
estimates
WP3 project leader: Bo Friis Nielsen
(DTU Informatics)
(1) Systematic biases in transport models (recognized ignorance), an empirical study
WP1 project leader: Bent Flyvbjerg (Oxford University)
DTU Transport, Kim Bang Salling 23
The Case Study: HH-Connection
• Connecting Denmark with Sweden: Scandinavian link
–Currently, close to the capacity limit on Oresund
HH-Connection
(alternatives)
Description
(Alignment of connection)
Cost
(million DKK)
Alternative 1 Tunnel for rail (2 tracks) person traffic only 7,700
Alternative 2 Tunnel for rail (1 track) goods traffic only 5,500
Alternative 3 Bridge for road and rail (2x2 lanes & 2 tracks) 11,500
Alternative 4 Bridge for road (2x2 lanes) 6,000
Note! 1 € 7.5 DKK
DTU Transport, Kim Bang Salling 24
The UNITE DSS Modelling Framework
The UNITE-DSS Decision Support Model for Risk Assessment
Determinstic Calculation
I) Cost-benefit analysis
Results: Point estimates in
terms of NPV, BCR, IRR
II) Optimism Bias Uplifts
Impact: Investment costs
Stochastic Calculation
III) Reference Class
Forecasting
Determination of Beta-PERT
distribution
Impact: Travel time savings
Results: Certainty graphs
and certainty values
Results: Point estimates in
terms of NPV, BCR, IRR
Determination of inputs to the
Beta-PERT distribution
IV) Reference Scenario
Forecasting
Determination of scenarios
and triple estimates
Impact: Travel time savings
Results: Certainty graphs
and values for scenarios
Trtiple estimate parameters
to the Beta-PERT distribution
DTU Transport, Kim Bang Salling 25
Deterministic Module – Entry data
DTU Transport, Kim Bang Salling 26
Results: Cost-Benefit Analysis
• Construction costs – by far the largest contributor of costs
• User Benefits – by far the largest contributor of benefits
– Consists of Ticket revenue and time savings
– Relies on the prognosis of future number of passengers i.e. demand forecasts
HH-Connection
(alternatives)
Cost
(million DKK)
BCR NPV
(million DKK)
Alternative 1 7,700 1.50 5,530
Alternative 2 5,500 0.16 -6,640
Alternative 3 11,500 2.71 28,240
Alternative 4 6,000 3.08 17,860
DTU Transport, Kim Bang Salling 27
Results : Optimism Bias Uplifts
• The BCR are lower, however, still point estimates towards DM
–Moreover an advanced form of sensitivity analysis
• Imply to introduce risk analysis and Monte Carlo simulation
HH-Connection
(alternatives)
Cost (uplifted)
(million DKK)
BCR (orig.)
(from slide 8)
BCR (uplifts):
80% uplift
Alternative 1 12,090 1.50 0.97
Alternative 2 8,640 0.16 0.10
Alternative 3 15,180 2.71 1.75
Alternative 4 7,920 3.08 1.98
DTU Transport, Kim Bang Salling 28
Stochastic module - @RISK
• The UNITE-DSS model is assigned an add-on software model named @RISK
• A range of distribution functions are shown
• ’Two’ non-parametric distrbutions have been tested/applied (green)
• ’Three’ parametric distributions have been tested/applied (orange)
DTU Transport, Kim Bang Salling 29
Input in UNITE-DSS – Construction cost
• Shape parameter k = 8 for road projects and k = 23 for rail projects (including air)
• The mean () and standard (std) deviation is calculated
• The scale parameter () is calculated on basis of the succesive principle
k
DTU Transport, Kim Bang Salling 30
Results (RCF): Monte Carlo simulation
DTU Transport, Kim Bang Salling 31
Reference Scenario Forecasting
• Accomodates scenario analysis and RCF
• Vertical regime: Economic development due to link
• Horizontal regime: Integration between borders
DTU Transport, Kim Bang Salling 32
Results from RSF
DTU Transport, Kim Bang Salling 33
The coupling of methodologies in achieving feasibility risk assessment
DTU Transport, Kim Bang Salling 34
Conclusions
• The UNITE-DSS model has been developed and functions as a flexible assessment tool applicable for wider risk oriented assessment for transport projects across different modes.
• The developed type of accumulated descending graph is found to be useful to inform about uncertainty relating to assessment of transport projects.
• Dependent on the information available parameter-based or parameter-free input probability distributions should be applied.
• It is possible to accommodate the recent results stemming from Optimism Bias and Reference Class Forecasting to produce relevant input to the PDFs for travel time savings and construction costs.
DTU Transport, Kim Bang Salling 35
Perspectives
• Investigation of introducing non-monetary aspects to the modelling framework as discussed in some of the papers is highly relevant
• Correlations between impacts are under review as to whether a general implementation is possible/needed
• The distinguishing between ”lack of knowledge” (uncertainty) and ”inherent randomness of the system” (variability) uncertainty should be investigated further
• Finally, the combinations of Optimism Bias and Risk Analysis needs further implementation – especially, the need for reference classes are obvious
DTU Transport, Kim Bang Salling
www.transport.dtu.dk/unite
• Large-scale investigation of uncertainties
• New up-to-date database information with regard to demand forecasts (and transport models)
• Involvement of researcher from Princeton and Oxford Universities
• Cross-disciplinarian research with practical applicability
36
DTU Transport, Kim Bang Salling 37
Thank you for listening!
Extra slides for presentation if needed
DTU Transport, Kim Bang Salling 39
Scenario Trend Development
Scenario Trend DevelopmentEconomic Growth and Level of Integration
90
100
110
120
130
140
150
160
2024 2029 2034 2039 2044 2049 2054 2059 2064 2069 2074
Years of evaluation
Inte
gra
tio
n level (I
nd
ex 1
00 in
2024)
High
Middle
Low
DTU Transport, Kim Bang Salling 40
Separation of Uncertainty
Nature of Uncertainty
Uncertainty (Epistemic):
Due to lack of Knowledge
Variability Uncertainty
(Ontological):
Due to inherent variability
within the system
Traditional aspects of modelling and policy
analysis:
- Limited and inaccurate data
- Measurement error
- Incomplete knowledge
- Limited uncerstanding
- Imperfect Models
- Subjective judgments
- Ambiguities
- etc.
Behavioural variability
(Micro)
Societal variability
(Meso & Macro)Natural randomness
DTU Transport, Kim Bang Salling 41
Cost-Benefit Analysis
Q’Q
P’’
E
B
A
P’
P
P
rice
- P
Quantity - QQ’’
''2
1''
2
1'
QQPPQQPPQPPB
TravellersGeneratedNewlyTravellersExisting
DTU Transport, Kim Bang Salling 42
Large changes in the Demand Curve
• ADT shift from 865 before to 10.000 after
• Cost per car before 300 DKK cost after 100 DKK
Demand curve: Cars (Øresund Fixed Link)
0
100
200
300
400
500
0 2000 4000 6000 8000 10000 12000
ADT
Co
sts
831,2 1045,4 PQPkQ
DTU Transport, Kim Bang Salling 43
Cost-Benefit Analysis
• Strengths:
– Transparency – all aspects are included in the analysis
– Comparable – Consistent, mostly due to the new manual
– Systematical data collection
• Weaknesses:
– ”False” sense of transparency – how to decide and undcover all aspects
– Practical measuring problem – models and unit prices
– Generations equity – same value today as last century
– Social equity (we are all a-like)
• Individual welfare
• Aggregation of individual welfare
DTU Transport, Kim Bang Salling 44
Dispute of criteria NPV vs. IRR
• As shown before the IRR expression is a polynomial equation with several roots
– Gradient and discount rate determines the choice
• IRR is independent from r
• NPV is dependent on r
• Hence, changing r to r* creates problems from the two projects suggested A and B.
IRR
NP
V
r r*
A B
IRRA IRRB
NP
VA
NP
VB
DTU Transport, Kim Bang Salling 45
Dispute about criteria NPV vs. BCR
• Given the system below with respectively costs and benefits for three system alternatives
• For a very short evaluation period of 1 year the NPV and B/C-rate are calculated
DTU Transport, Kim Bang Salling 46
Public vs. Private
• Tax distortion of 1.2 is introduced due to the financing of projects through taxes:
– E.g. Person A willing to perform a job for 100 DKK
– Person B is willing to pay to get the job done for 110 DKK
– 50% tax would endure that Person B would pay 55 DKK
– Society loses the actual surplus of 10 DKK
• Net Taxation factor is introduced of 1.17:
– Since we operate with market prices, a private company would endure duties, taxes etc. on commodities
– The State obviously does not have to pay that
– 17% has been found as an average
DTU Transport, Kim Bang Salling 47
Research Outcomes
DTU Transport, Kim Bang Salling 48
Full scale uplifts from COWI and Flyvbjerg
DTU Transport, Kim Bang Salling 49
Beta Distribution
• Typically parameterized by two shape parameters [, ]:
DTU Transport, Kim Bang Salling 50
Gamma Distribution
• Typically parameterized by a shape and scale parameters [k, ]:
xotherforxfandkxex
k
kxf kxk
k
0)( ....4,3,2,0,!1
)( 1
kiancethewhilek
k
k 1var var
1
DTU Transport, Kim Bang Salling 51
Succesive Calculation
Post Beskrivelse Mængde Enhed a b C m s varians*10-6
1 Opstartsarbejde 1 stk. 37.500 187.500 450.000 210.000 82.500 6.806
2 Boldbaner 50.000 m2 30 75 120 3.750.000 900.000 810.000
3 Andre græsarealer 25.000 m2 8 15 30 412.500 112.500 12.656
4 Parkanlæg 20.000 m2 8 23 60 540.000 210.000 44.100
5 Befæstede arealer 15.000 m2 90 225 330 3.285.000 720.000 518.400
6 Afsluttende arbejde 1 stk. 37.500 150.000 375.000 172.500 67.500 4.556
7 Generelle forhold 8.370.000 Sum -10 % 0 % 20 % 167.400 502.200 252.205
Kalkuleret middelværdi 8.537.400 1.648.724
Tilhørende spredning, beregnet som kvadratroden af
summen af variansen
1.284.026
Post Beskrivelse Mængde Enhed a b c m s varians*10-6
1 Opstartsarbejde 1 stk. 37.500 187.500 450.000 210.000 82.500 6.806
2 Boldbaner 50.000 m2 3.006.000 5.234
2.1 Rydning og afretning 50.000 m2 11,25 12,3 13,35 615.000 21.000 441
2.2 Dræn 50.000 m2 14,7 16,5 18,75 829.000 40.500 1.640
2.3 Vandingssystem 50.000 m2 9,75 12,75 13,5 615.000 37.500 1.406
2.4 Muld og planering 50.000 m2 12 13,5 15,9 684.000 39.000 1.521
2.5 Såning 50.000 m2 4,5 5,25 6 262.500 15.000 225
3 Andre græsarealer 25.000 m2 7,5 15 30 412.500 112.500 12.656
4 Parkanlæg 20.000 m2 7,5 22,5 60 540.000 210.000 44.100
5 Befæstede arealer 15.000 m2 90 225 330 3.285.000 720.000 518.400
6 Afsluttende arbejde 1 stk. 37.500 150.000 375.000 172.500 67.500 4.556
7 Generelle forhold 7.626.000 sum -10 % 0 % 20 % 152.520 457.560 209.361
Kalkuleret middelværdi 7.778.520 801.114
Tilhørende spredning, beregnet som kvadratroden af summen af
variansen
895.050
DTU Transport, Kim Bang Salling 52
Data fitting
• The data fits are conducted by Maximum likelihood estimators:
– Estimates the distribution parameters
– Maximum likelihood parameter estimation is to determine the parameters that maximize the probability (likelihood) of the sample data
• The goodness of fits interpreted by using Chi-squared [2] statistics:
– The sum of differences between observed and expected outcomes
– where O is an observed outcome
– and E is an expected frequency
E
EO2
2
DTU Transport, Kim Bang Salling 53
Background literature (international)
2002 2003 2004
DTU Transport, Kim Bang Salling 54
Background literature (National)
2007 2007 2008
DTU Transport, Kim Bang Salling 55
Back et al. (2000)
• Four bullet points for estimating construction costs with probability distributions have been proposed in:
– Upper and lower limits which ensures that the analyst is relatively certain values does not exceed. Consequently, a closed-ended distribution is desirable.
– The distribution must be continuous
– The distribution will be unimodal; presenting a most likely value
– The distribution must be able to have a greater freedom to be higher than lower with respect to the estimation – skewness must be expected.
DTU Transport, Kim Bang Salling 56
Composite Model for Assessment
Alt. 1
Alt. 2
Alt. 3
CBA
MCA
.
.
.
.
.
A
B
C
D
.
.
.
SMART
AHP
B/C
DTU Transport, Kim Bang Salling 57
A Brief History
• 1950’s: Introduction of CBA in USA – Highway’s connecting East-West
• 1960’s: CBA Methodology reaches Europe – New Motorway Schemes
• 1970’s: Traditional traffic impacts are introduced
• 1980’s: The methodology reaches Denmark together with widespread impacts within the Multi-Criteria methodology
• 1990’s: Full implementation in Denmark a general acceptance of CBA & MCA
• 2003: The Danish Ministry of Transport published in 2003 a guideline for making socio-economic analysis in the Danish Transport Sector