using npmrds: lessons learned
TRANSCRIPT
![Page 1: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/1.jpg)
Using NPMRDS: Lessons Learned
Kartik KaushikCenter for Advanced Transportation Technologies
University of Maryland
![Page 2: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/2.jpg)
Lesson 1: Accessing
2
![Page 3: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/3.jpg)
Downloading
• Need to sign data use agreement with HERE Inc. to get access
• Coordinated through email• Total archive size is roughly 5 to 7 GB per month (packed)
• For the whole country, all files• States are grouped in zones, with an archive for each zone
• 700 MB per zone on average (packed)• Downloading using browser is not recommended (even for zonal archives)
• Use of FTP is recommended• Winscp is a good FTP client for Windows• Can be scripted to automatically check and download updates each month 3
![Page 4: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/4.jpg)
Files Overview
• Three basic types of files• Data files only contain travel times for each record
• Join with TMC definition file is necessary to obtain speeds• File changes each month, to reflect the month’s observations• Data file size depends on the size of state – few 100 MB to 1‐3 GB per month (unpacked)
• TMC definition file contains information to identify a segment
• Contains the lengths of each TMC segment• Changes only when TMC table gets updated (updates released quarterly)
• Packaged with each month’s data – about 4 MB in size (unpacked)• Shapefile only contains sub‐TMC segments
• Join with lookup table, TMC definition file, and Data files is required to visualize data on map
• Has update/release cycle of 6 months• About 2 GB in size (unpacked) 4
![Page 5: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/5.jpg)
Opening and Viewing Files
• Microsoft cannot help• Cannot use Excel, or Access because the files are too large
• Need core database programs or scripting programs adept to large text files
• MySQL is good for beginning – storing, viewing, querying• PostgreSQL is more advanced – can perform lot more analysis, and visualizations
• SAS, SPSS, etc. are also good commercial alternatives• Programming languages like Python, R, Matlab, etc. can also be used, especially if data needs to be analyzed
• To open and view the included Shapefile, ArcGIS is required
• Can also use PostGIS (extends PostgreSQL), but is a little complicated
5
![Page 6: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/6.jpg)
Analysis and Visualization
• MySQL for basic analysis – descriptive statistics and the like• PostgreSQL and ArcGIS for advanced analysis
• PostgreSQL has extensions to allow programming languages to connect
• ArcGIS has embedded Python and Visual Basic, in addition to model builder
• For detailed analysis Python with Pandas, R or Matlab is recommended
• Analysis includes computing performance measures: PTI, TTI, BTI, etc.
• For graphical visualizations Python, R or Matlab are recommended
• One can also import the results to Excel for graphical visualizations• Easiest to use ArcGIS to visualize on a map
• If using PostgreSQL, download supplemental GIS package: PostGIS
6
![Page 7: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/7.jpg)
Lesson 2: Data
7
![Page 8: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/8.jpg)
Traffic Message Channel (TMC) codes• TMC code references to a particular segment on the road
• However, TMC information contained within the static file is limited
• Local name and one random coordinate point is given• Probably use the provided shapefile to extend the geo information within the static files
• Can get accurate start and end coordinates from the shapefile (but not easy)
• Ordering obtained coordinate points by road direction can be a nightmare
• Also, caution is advised when comparing TMC definitions from multiple different vendors
8
![Page 9: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/9.jpg)
Example of limited info about TMC (110N06799)
TMC information in Static File TMC information in Shapefile
• Joining the static file to the shapefile provides a full picture• Joining may be problematic due to many‐to‐many relationship between the shapefile and
TMC static file
• Lat/Long point in the static file may be begin, end or somewhere in between on a segment
1.06937 mi, Eastbound, Lee Hwy, US‐29, Fairfax, VA,
USALee Hwy, From
Reference Node, Not a Ramp, No frontage to
property, No contra‐flow
9
![Page 10: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/10.jpg)
Travel Time Data
• Direct calculation of traditional performance measures (TTI, PTI, and BTI) is problematic
• Data is missing for some reporting intervals• If no base data, no traffic record is provided• Creates sparse datasets• Complicates outlier filtering and smoothing
• Data is unfiltered, and noisy• Causes problems when trying to impute missing data
• Other methods of processing data are recommended• Methods that do not require imputation of data works best• Methods that can do without outlier filtering enable small and easy algorithms
10
![Page 11: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/11.jpg)
Lesson 3: Analysis
11
![Page 12: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/12.jpg)
Overlaying Data
• Originally developed at CATT to deal with issues presented by interrupted flow facilities (arterials)
• Involves repeatability of traffic conditions and characteristics over days or months
• Overlay data from those days where repeated traffic is expected
• Weekday data from a couple of weeks (used in this presentation)
• Only Monday, Tuesday… data for a month or two• Any other similar logical grouping
• Overlay size can be adjustable, but 24 hour periods is recommended
12
![Page 13: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/13.jpg)
Example of Sparse Data:NPMRDS data for one day, one random TMC
13
![Page 14: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/14.jpg)
Result of Overlaying Data:Same TMC, 2 weeks’ worth of weekday data
14
![Page 15: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/15.jpg)
Computing Performance
• Performance is computed using 24 hour overlay scatter plots
• Slices of data from the overlay plots is taken• Used here is 1 hour windows, also recommended (unless overlaid data is also sparse)
• Percentiles of travel time within the slice is obtained• 5% increments are used here, going from 5% to 95%, (recommended)
• Percentiles ensure that one or two outliers have minimal impact on results (because of the nature of percentile computation)
• Percentiles are then graphed to create Cumulative Distribution Functions (CDF)
• Performance measures can be computed from CDFs15
![Page 16: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/16.jpg)
Lesson 4: Case Studies
16
![Page 17: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/17.jpg)
Validation with VPP and BTM data
• Results obtained from NPMRDS was compared against Vehicle Probe Project (VPP) and Bluetooth Traffic Monitoring (BTM) data
• BTM is obtained using portable Bluetooth sensors developed by UMD
• BTM is raw observations of actual vehicle traversals• Like NPMRDS, only observed vehicles are reported
• VPP is commercial probe data, available through contracts awarded by I‐95 Corridor Coalition
• VPP is probe data similar to NPMRDS, but is a real time feed at 1 minute intervals
• VPP is smoothed, filtered and imputed• Two case studies are presented here: a freeway‐ish corridor, and an arterial with heavily interrupted flow
17
![Page 18: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/18.jpg)
VA‐7 in Virginia: Freeway‐ish4 Lanes, 1 signal, 0.88 miles
18
![Page 19: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/19.jpg)
24 Hour Scatter Plots
NPMRDS(All Vehicle)
VPP
BTM
19
![Page 20: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/20.jpg)
Peak Hour: 5 PM to 6 PM (BTM, VPP, NPMRDS)
NPMRDS(All Vehicle)
VPP
BTM
0 5 10 15 20 250
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Travel Time - Minutes
Percen
tile
Travel Time CFD Diagram
12AM 3AM 6AM 9AM 12PM 3PM 6PM 9PM 12AM0
5
10
15
20
25
Hour of Day 0-24
Trav
el Time - M
inutes
Hourly Overlay Scatterplot
20
![Page 21: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/21.jpg)
Peak Hour: 5 PM to 6 PM (NPMRDS Only)
All Vehicle
Passenger Car
Freight Truck
21
![Page 22: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/22.jpg)
US‐1 in NJ: Interrupted Flow Arterial
4 Lanes, 6 signals, 1.37 miles
22
![Page 23: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/23.jpg)
24 Hour Scatter Plots
NPMRDS(All Vehicle)
VPP
BTM
23
![Page 24: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/24.jpg)
Peak Hour: 4 PM to 5 PM (BTM, VPP, NPMRDS)
NPMRDS(All Vehicle)
0 2 4 6 8 100
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Travel Time - Minutes
Per
cent
ile
Travel Time CFD Diagram
12AM 3AM 6AM 9AM 12PM 3PM 6PM 9PM 12AM0
1
2
3
4
5
6
7
8
9
10
Hour of Day 0-24
Trav
el T
ime - Minut
es
Hourly Overlay Scatterplot
VPP
BTM
24
![Page 25: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/25.jpg)
Peak Hour: 4 PM to 5 PM (NPMRDS Only)
All Vehicle
Passenger Car
Freight Truck
25
![Page 26: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/26.jpg)
Take Aways
• Accessing and Analyzing• Data size too large for Microsoft Office• Need database or other specialized analysis packages• GIS tools for visualizing on a map
• Comparison with probe and Bluetooth data• 24 hour overlay plots and corresponding CDF curves capture recurring congestion
• NPMRDS tends to capture variance in traffic similar to Bluetooth, but still subject to same limitations of probe data
• The overlay and percentile method can be used to calculate performance measures without imputation or outlier filtering
26
![Page 27: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/27.jpg)
Thank YouQuestions?
![Page 28: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/28.jpg)
AppendixNot to be presented, unless to answer questions.
28
![Page 29: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/29.jpg)
Example of TMC mismatch between vendors • NPMRDS TMC definition vs. INRIX (TomTom) TMC definition
110P04418 NPMRDS1.332 miles
110P04418 INRIX 1.09 miles
110+04419 INRIX 1.97 miles
110P+04418 INRIX 0.60 miles
29
![Page 30: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/30.jpg)
Additional Corridor
30
![Page 31: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/31.jpg)
Route 29 in Virginia: Freeway‐ish4 Lanes, 2 signals, 1.07 miles
31
![Page 32: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/32.jpg)
24 Hour Scatter Plots
NPMRDS(All Vehicle)
VPP
BTM
32
![Page 33: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/33.jpg)
Peak Hour: 7 AM to 8 AM (BTM, VPP, NPMRDS)
NPMRDS(All Vehicle)
VPP
BTM
0 2 4 6 8 100
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Travel Time - Minutes
Percen
tile
Travel Time CFD Diagram
12AM 3AM 6AM 9AM 12PM 3PM 6PM 9PM 12AM0
1
2
3
4
5
6
7
8
9
10
Hour of Day 0-24
Trav
el Tim
e - M
inutes
Hourly Overlay Scatterplot
33
![Page 34: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/34.jpg)
Peak Hour: 7 AM to 8 AM (NPMRDS Only)
All Vehicle
Passenger Car
Freight Truck
34
![Page 35: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/35.jpg)
Compute Performance Measures from CDF
35
![Page 36: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/36.jpg)
Formulae for Performance Measures• Travel Time Index (TTI) =
• Or, TTI = ∑
(because, mean = ∑ from
percentiles)
• Planning Time Index (PTI) =
• Buffer Time Index (BTI) =
• Or, BTI = ∑
• Inter‐Quartile Range (IQR) =
36
![Page 37: Using NPMRDS: Lessons Learned](https://reader034.vdocuments.mx/reader034/viewer/2022051306/62799f1e03d7330dd67403bf/html5/thumbnails/37.jpg)
Method
37
TTI
IQRPTI
BTI