cctv data migration project - cnam.cacnam.ca/wordpress/wp-content/uploads/pediadocs/d3... · cctv...
TRANSCRIPT
CCTV Data Migration Project
Charlotte WyattAlex Wojcik
Courtney MacMullenGIS Technicians, SEG Consulting
About the Project• Planning, prioritizing inspection areas• Accessibility and maintenance of data• Manual data entry
About UsSimplifying Complex Information is What We Do
• Automation
• Usable data formats
Project Scope• Extract data from CCTV pipe inspection reports• Process ~1,800 PDFs within three months• Develop an automated solution
Survey Data
Site Data
Pipe Data
Observation Data
Condition Ratings
ApproachGIS provides a central location to store, manage,
and disseminate data
GIS for Sewer Data• Central location for asset data• Increased accessibility of CCTV data• Access to condition ratings, inspection dates• Monitoring of asset lifecycle and cost
Methodology• Python is a powerful programming language• Leverage ArcPy (interaction with Python and ArcGIS)• Third party modules augment automation• Loop through PDFs to extract data• Migrate into optimized geodatabase
1,800 PDF
Reports
Define Report
Template
Extract Report
Components
Survey Data
(line)
Validate
(QC)
Observation
Data (points)
Define Report Style• Report layout varies• Inconsistent location of text• Trial and error testing
1,800 PDF
Reports
Define Report
Template
Extract Report
Components
Survey Data
(line)
Validate
(QC)
Observation
Data (points)
Text CoordinatesDetermine text coordinates within PDF using GhostScript
1,800 PDF
Reports
Define Report
Template
Extract Report
Components
Survey Data
(line)
Validate
(QC)
Observation
Data (points)
Add Data to Layers• Attribute table for point feature class:
1,800 PDF
Reports
Define Report
Template
Extract Report
Components
Survey Data
(line)
Validate
(QC)
Observation
Data (points)
Linear Referencing• Find matching manhole IDs to locate pipe feature• Place observation points along the pipe line• Give incident data a spatial reference
Linear Referencing
Symbology• Observation points symbolized according
to NASSCO PACP defect codes• Direction of survey indicated by arrows
1,800 PDF
Reports
Define Report
Template
Extract Report
Components
Survey Data
(line)
Validate
(QC)
Observation
Data (points)
Recommendations• Ensure validity of manhole IDs• Create consistent report template structure• Save edited reports as text-based PDFs
The Solution• Increases accessibility of data• Reduces manual data entry• Improves cost-efficiency and productivity• Promotes monitoring and maintenance of assets
Tying it all Together• What does a PDF report look like once it is in GIS?• How would an Asset Manager use the data?
Fraser St July23 2010.pdf
Dartmouth, NS
Header Fraser St July23 2010.pdf
Footer
Pipe Data in Report
… becomes associated to line feature
Pipe Data in GIS
Condition Ratings
Upstream MHCB14054
Downstream MHCB14055
Attribute Table:
CCTV Data in Report
… becomes associated to point feature
CCTV Data in GIS
First observation at0.0 metres
Attribute Table:
1.74 m3.47 m
0 m
Second observation at1.74 metres
1.74 m3.47 m
0 m
CCTV Data in GISAttribute Table
Using the Data
Fields can be sortedE.g.: Pipes sorted by condition rating or by date surveyed
SQL queries can be made on fields to find specific dataGIS affords an added level of intelligence, spatial analysis is possible
Search by PDF name
Search by street
Search for pipes that have a Quick Structural Rating not equal to 0
Search by manhole ID
Using the Data
Results93.4% of PDF reports were successfully processed
• 11,317 pipe lines• 92,610 observation points
% of records “on the map”:• 81.2% Pipe lines• 85.7% Observation
points
Results
“Missing” records:• No matching manholes
• Quantity and comprehensiveness of data obtained• Time efficient solution:
• Three months of script developmentvs. years of traditional manual data entry
• Great return on investment• Data now exists in GIS format!
Achievements
Courtney MacMullenSEG ConsultingGIS [email protected]
Alexander WojcikSEG ConsultingGIS [email protected]
Charlotte WyattSEG ConsultingGIS [email protected]
Questions