how good are our models?
TRANSCRIPT
![Page 1: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/1.jpg)
1 May 2023 AITPM Conference Sydney 2016
How good are our models?Grant Smith and David Hunter
![Page 2: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/2.jpg)
1 May 2023 AITPM Conference Sydney 2016
Wollongong/Shellharbour
![Page 3: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/3.jpg)
1 May 2023 AITPM Conference Sydney 2016
Wollongong/Shellharbour
253,000 people96,300 households85,300 jobs
![Page 4: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/4.jpg)
1 May 2023 AITPM Conference Sydney 2016
Model History
1993- First model built using 1991 dataNo future years
![Page 5: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/5.jpg)
1 May 2023 AITPM Conference Sydney 2016
Model History
1998 – two models built using 1996 dataOne with 376 zones
Future years of 2006 and 2016
![Page 6: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/6.jpg)
1 May 2023 AITPM Conference Sydney 2016
Model History
And one at 1996 with 470 zonesFuture year of 2026
![Page 7: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/7.jpg)
1 May 2023 AITPM Conference Sydney 2016
Model History
2004 – model updated to 2001Experiment with dynamic and four step models
No future years
![Page 8: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/8.jpg)
1 May 2023 AITPM Conference Sydney 2016
Model History
In 2010 the model updated to 2006 censusFuture years of 2011, 2021, 2026 and 2036
![Page 9: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/9.jpg)
1 May 2023 AITPM Conference Sydney 2016
SummaryFive models
1991 1996 (376 zones) 1996 (470 zones) 2001 2006
![Page 10: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/10.jpg)
1 May 2023 AITPM Conference Sydney 2016
DifferencesModel Detail
Number of zones
![Page 11: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/11.jpg)
1 May 2023 AITPM Conference Sydney 2016
DifferencesNumber of zones
1991 1996 1996 2001 20060
100
200
300
400
500
600
Zones
![Page 12: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/12.jpg)
1 May 2023 AITPM Conference Sydney 2016
DifferencesModel Detail
Number of zones Number of links
![Page 13: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/13.jpg)
1 May 2023 AITPM Conference Sydney 2016
1991 1996 1996 2001 20060
5000
10000
15000
20000
25000
Links
DifferencesNumber of links
![Page 14: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/14.jpg)
1 May 2023 AITPM Conference Sydney 2016
10.0km
Network 19917,839 links
![Page 15: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/15.jpg)
1 May 2023 AITPM Conference Sydney 2016
10.0km
Network 1996376 zone system6,009 links
![Page 16: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/16.jpg)
1 May 2023 AITPM Conference Sydney 2016
Network 1996470 zone system12,569 links
![Page 17: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/17.jpg)
1 May 2023 AITPM Conference Sydney 2016
Network 200112,508 links
![Page 18: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/18.jpg)
1 May 2023 AITPM Conference Sydney 2016
50.0km
Network 200619,506 links
![Page 19: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/19.jpg)
1 May 2023 AITPM Conference Sydney 2016
DifferencesModel technology
1991 Generation – 3x3 category model (employees and cars per
HH) Distribution – negative exponential of cost Central area Parking model included Assignment – intersection delays at the approach
![Page 20: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/20.jpg)
1 May 2023 AITPM Conference Sydney 2016
DifferencesModel technology
1996 (376 zone) Generation – 3x3 category model (employees and cars per
HH) Distribution – negative exponential of cost Central area Parking model included Assignment – intersection delays at the approach
![Page 21: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/21.jpg)
1 May 2023 AITPM Conference Sydney 2016
DifferencesModel technology
1996 (470 zone) Generation – 3x3 category model (employees and cars per
HH) Distribution – negative exponential of cost Central area Parking model included Assignment – intersection delays at turn movement level
![Page 22: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/22.jpg)
1 May 2023 AITPM Conference Sydney 2016
DifferencesModel technology
2001 Generation – 3x3 category model (employees and cars per
HH) Distribution – negative exponential of cost Central area Parking model included Assignment – intersection delays at turn movement level
![Page 23: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/23.jpg)
1 May 2023 AITPM Conference Sydney 2016
DifferencesModel technology
2006 Generation – 5x3 category model (persons and cars per HH) Distribution – negative exponential of time Central area Parking model included Assignment – intersection delays at turn movement level
- Multi class assignment
![Page 24: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/24.jpg)
1 May 2023 AITPM Conference Sydney 2016
Model Validation
1 4 7 10 13 16 19 22 25 28 31 340
5
10
15
20
25
30
35
Number of screenline comparisons
1996 (470)1
2006
GEH
1996 (376)
1991
2001
![Page 25: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/25.jpg)
1 May 2023 AITPM Conference Sydney 2016
South of Bulli Pass
South of Mount Ousley
South of Masters Road
South of Northcliffe Road
South of Dapto
Escarpment
Comparisonscreenlines
![Page 26: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/26.jpg)
1 May 2023 AITPM Conference Sydney 2016
2006 Forecast from the 1996 model
Linear interpolation between 1996 and 2016 470 zones in the 2006 model compared with 376 More detailed network (twice as many links) Different generation and distribution models Intersections modelled at movement rather than
approach level in the 2006 model
![Page 27: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/27.jpg)
1 May 2023 AITPM Conference Sydney 2016
Comparison at 20062006 Comparisons
2006 Model 1996 Model Difference
Households 92,609 101,875 +10%
Jobs 77,696 82,092 +6%
Trips 57,842 70,092 +21%
Screenlines
South Bulli Pass 4,926 5,678 +15.3%
Escarpment 4,970 5,636 +13.4%
South of Mt Ousley 11,059 10,081 +9.7%
South of Masters 12,862 14,634 +13.8%
South of Northcliffe 9,981 12,987 +30.1%
South of Dapto 7,423 8,645 +16.4%
Railway 35,143 40,736 +15.9%
![Page 28: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/28.jpg)
1 May 2023 AITPM Conference Sydney 2016
Two Models validated in 1996
Same land use data 470 zones compared with 376 More detailed network (twice as many links in the 470
zone model) Intersections modelled at movement rather than
approach level
![Page 29: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/29.jpg)
1 May 2023 AITPM Conference Sydney 2016
Difference between two modelsBoth using 1996 land use
1996 Comparisons
470 zone model 376 zone model Difference
Households 85,038 83,092 +2.3%Jobs 68,040 72,684 -11.2%Trips 55,784 56,889 -2.0%Trip length (minutes) 12.6mins 11.2mins +12.5%Screenlines South Bulli Pass 4,086 4,430 -8.4% Escarpment 4,037 4,175 -3.4% South of Mt Ousley 8,194 9,130 -11.4% Figtree South of Masters 11,374 11,439 -0.6% South of Northcliffe 9,789 9,332 +4.7% South of Dapto 7,159 6,761 +5.6% Railway 30,276 30,380 -0.3%
![Page 30: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/30.jpg)
1 May 2023 AITPM Conference Sydney 2016
Model Validation
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 820
5
10
15
20
25
30
Number of points
1996 (470)1
GEH
1996 (376)
![Page 31: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/31.jpg)
1 May 2023 AITPM Conference Sydney 2016
Common land use over the five models
2011 census data converted into the zone system used by each model
![Page 32: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/32.jpg)
1 May 2023 AITPM Conference Sydney 2016
Common land use over the five models
2011 census data converted into the zone system used by each model
Models run and converged using the software versions used for the original validation
![Page 33: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/33.jpg)
1 May 2023 AITPM Conference Sydney 2016
Five models using 2011 data
2011 land use applied to the various models
2006 model 2001 model 1996 470 zone model
1996 376 zone model 1991 model
Households 96,354 96,354 96,354 96,144 94,953Jobs 85,307 85,311 85,311 85,050 87,792Cars 158,524 158,523 158,523 158,090 155,785Trips 65,501 67,518 67,368 74,104 77,073Trip length (minutes) 11.40 mins 13.95mins 13.81mins 11.64mins 11.66minsScreenlines South Bulli Pass 5,609 5,616 5,468 5,844 4,242 Escarpment 5,735 5,655 5,675 5,649 3,210 South of Mt Ousley 11,069 10,696 10,722 11,493 11,048 South of Masters 13,219 12,507 12,167 13,009 12,443 South of Northcliffe 10,358 11,006 10,985 11,454 10,746 South of Dapto 8,077 8,540 8,523 8,911 8,108 Railway 37,145 37,744 37,303 38,416 40,607
![Page 34: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/34.jpg)
1 May 2023 AITPM Conference Sydney 2016
Five models using 2011 data
2011 land use applied to the various models
2006 model 2001 model 1996 470 zone model
1996 376 zone model 1991 model
Households 96,354 96,354 96,354 96,144 94,953Jobs 85,307 85,311 85,311 85,050 87,792Cars 158,524 158,523 158,523 158,090 155,785Trips 65,501 67,518 67,368 74,104 77,073Trip length (minutes) 11.40 mins 13.95mins 13.81mins 11.64mins 11.66minsScreenlines South Bulli Pass 5,609 5,616 5,468 5,844 4,242 Escarpment 5,735 5,655 5,675 5,649 3,210 South of Mt Ousley 11,069 10,696 10,722 11,493 11,048 South of Masters 13,219 12,507 12,167 13,009 12,443 South of Northcliffe 10,358 11,006 10,985 11,454 10,746 South of Dapto 8,077 8,540 8,523 8,911 8,108 Railway 37,145 37,744 37,303 38,416 40,607
![Page 35: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/35.jpg)
1 May 2023 AITPM Conference Sydney 2016
Comparison of screenlinesSouth Bulli Pass
2011 forecast flows for each model relative to the 2006 assignment
2001 model 1996 470 zone model
1996 376 zone model 1991 model
South Bulli Pass
Assigned flow 5,616 5,468 5,844 4,242
Percent difference 0% -3% 4% -24%
Correlation coefficient 1.00 1.00 1.00 1.00
% RMS 8.25 14.63 5.94 38.92
R2 1.00 1.00 1.00 1.00
GEH overall 0.1 1.9 3.1 19.5
% GEH <7.0 100% 75% 100% 25%
![Page 36: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/36.jpg)
1 May 2023 AITPM Conference Sydney 2016
Comparison of screenlinesEscarpment
2011 forecast flows for each model relative to the 2006 assignment
2001 model 1996 470 zone model
1996 376 zone model 1991 model
Escarpment
Assigned flow 5,655 5,675 5,649 3,210
Percent difference -1% -1% -1% -44%
Correlation coefficient 0.990 0.989 0.992 0.970
% RMS 20.84 22.35 18.0 87.96
R2 0.980 0.978 0.984 0.940
GEH overall 1.1 0.8 1.1 37.8
% GEH <7.0 66.7% 83% 75% 67%
![Page 37: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/37.jpg)
1 May 2023 AITPM Conference Sydney 2016
Comparison of screenlinesSouth of Mt Ousley
2011 forecast flows for each model relative to the 2006 assignment
2001 model 1996 470 zone model
1996 376 zone model 1991 model
South of Mt Ousley
Assigned flow 10,696 10,722 11,493 11,048
Percent difference -3% -3% 4% 0%
Correlation coefficient 0.974 0.995 0.939 0.838
% RMS 13.99 9.15 20.92 33.28
R2 0.950 0.991 0.882 0.703
GEH overall 3.1 3.3 4.0 0.2
% GEH <7.0 80% 50% 60% 30%
![Page 38: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/38.jpg)
1 May 2023 AITPM Conference Sydney 2016
Comparison of screenlinesFigtree South of Masters
2011 forecast flows for each model relative to the 2006 assignment
2001 model 1996 470 zone model
1996 376 zone model 1991 model
Figtree South of Masters
Assigned flow 12,507 12,167 13,009 12,443
Percent difference 6% 3% 10% 6%
Correlation coefficient 0.983 0.982 0.996 0.965
% RMS 23.09 20.46 26.08 30.63
R2 0.966 0.965 0.992 0.913
GEH overall 6.6 3.6 11.1 6.1
% GEH <7.0 40% 56% 44% 11%
![Page 39: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/39.jpg)
1 May 2023 AITPM Conference Sydney 2016
Comparison of screenlinesSouth of Northcliffe
2011 forecast flows for each model relative to the 2006 assignment
2001 model 1996 470 zone model
1996 376 zone model 1991 model
South of Northcliffe
Assigned flow 11,006 10,985 11,454 10,746
Percent difference 6% 6% 11% 4%
Correlation coefficient 1.00 1.00 0.989 0.988
% RMS 16.63 18.23 31.02 36.56
R2 0.999 1.00 0.979 0.975
GEH overall 6.3 6.1 10.5 3.8
% GEH <7.0 50% 50% 38% 63%
![Page 40: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/40.jpg)
1 May 2023 AITPM Conference Sydney 2016
Comparison of screenlinesSouth of Dapto
2011 forecast flows for each model relative to the 2006 assignment
2001 model 1996 470 zone model
1996 376 zone model 1991 model
South of Dapto
Assigned flow 8,540 8,523 8,911 8,108
Percent difference 6% 6% 10% 0%
Correlation coefficient 0.997 0.992 0.998 0.849
% RMS 14.67 17.59 19.38 43.6
R2 0.994 0.983 0.996 0.720
GEH overall 5.1 4.9 9.0 .3
% GEH <7.0 50% 88% 88% 50%
![Page 41: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/41.jpg)
1 May 2023 AITPM Conference Sydney 2016
Comparison of screenlinesRailway
2011 forecast flows for each model relative to the 2006 assignment
2001 model 1996 470 zone model
1996 376 zone model 1991 model
Railway
Assigned flow 37,744 37,303 38,416 40,607
Percent difference 2% -1% 3% 9%
Correlation coefficient 0.935 0.938 0.959 0.908
% RMS 43.97 42.22 34.98 51.61
R2 0.874 0.879 0.919 0.825
GEH overall 3.1 1.9 6.5 17.6
% GEH <7.0 54% 57% 53% 49%
![Page 42: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/42.jpg)
1 May 2023 AITPM Conference Sydney 2016
Conclusions
Most models produced similar trip generation rates.
![Page 43: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/43.jpg)
1 May 2023 AITPM Conference Sydney 2016
Conclusions
Most models produced similar trip generation rates. As detail increased, validation statistics improved. The
2006 validation was extremely good
![Page 44: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/44.jpg)
1 May 2023 AITPM Conference Sydney 2016
Conclusions
All household forecasts were high
![Page 45: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/45.jpg)
1 May 2023 AITPM Conference Sydney 2016
Conclusions
All household forecasts were high When the same land use is used, all the models are
consistent at the overall screenline level
![Page 46: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/46.jpg)
1 May 2023 AITPM Conference Sydney 2016
Conclusions
All household forecasts were high When the same land use is used, all the models are
consistent at the overall screenline level The coarser models are not good at getting the
distribution on roads within screenlines correct
![Page 47: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/47.jpg)
1 May 2023 AITPM Conference Sydney 2016
Conclusions
The finer the zone system and the more detail that can be included in the network, the better the validation
![Page 48: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/48.jpg)
1 May 2023 AITPM Conference Sydney 2016
Conclusions
The finer the zone system and the more detail that can be included in the network, the better the validation
Forecasts of travel are only as good as the land use data that is used as input.
![Page 49: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/49.jpg)
1 May 2023 AITPM Conference Sydney 2016
Conclusions
The finer the zone system and the more detail that can be included in the network, the better the validation
Forecasts of travel are only as good as the land use data that is used as input.
If the land use data is accurate, then the forecasts of traffic flow will also be accurate
![Page 50: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/50.jpg)
1 May 2023 AITPM Conference Sydney 2016
Conclusions
The finer the zone system and the more detail that can be included in the network, the better the validation
Forecasts of travel are only as good as the land use data that is used as input.
If the land use data is accurate, then the forecasts of traffic flow will also be accurate
a high degree of confidence can be placed in the models as forecasting tools
![Page 51: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/51.jpg)
1 May 2023 AITPM Conference Sydney 2016
….and finally
He who lives by a crystal ball rapidly learns to eat ground glass
![Page 52: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/52.jpg)
1 May 2023 AITPM Conference Sydney 2016
….and finally
He who lives by a crystal ball rapidly learns to eat ground glass
When you make a prediction it will always be wrong – the only questions are when and by how much
![Page 53: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/53.jpg)
1 May 2023 AITPM Conference Sydney 2016
….and finally
He who lives by a crystal ball rapidly learns to eat ground glass
When you make a prediction it will always be wrong – the only questions are when and by how much
When you are right with a projection – don’t let anyone forget it
![Page 54: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/54.jpg)
1 May 2023 AITPM Conference Sydney 2016
Acknowledgments
Wollongong City Council Shellharbour City Council Transport for New South Wales (Roads and Marine
Services)
![Page 55: How good are our models?](https://reader031.vdocuments.mx/reader031/viewer/2022011721/587894011a28ab375f8b6373/html5/thumbnails/55.jpg)
1 May 2023 AITPM Conference Sydney 2016
Thank you