automating regional data integration with python & arcpy (heather widlund)

AUTOMATING REGIONAL DATA INTEGRATION WITH PYTHON & ARCPY

Heather WidlundGIS Coordinator, San Miguel County

West Region Data Integration Group

Integrated dataStandardized schemasoStreet centerlines

oAddress/structure points

oAddressed places/landmarks

Other: ESNs, PSAP boundaries, Parcels

Emergency preparedness & response focus oVerify emergency notification addresses (CodeRED)

oVisual mapping in multi-agency dispatch

o911 call address verification (Spillman CAD)

4/29/2016GIS COLORADO | SPRING 2016 MEETING | BRECKENRIDGE, COLORADO


Data updates are arduous.What is the problem?

Why is this a problem?

Data updates are infrequent.

What are the consequences?

What is the solution?


>>Make a machine do the work:

. . .

Automation ProjectPhases

1) Source data with valid schema2) Ability to integrate easily & automatically3) Data updates pushed to Spillman CAD


Script tools, Python & arcpy libraryoFamiliar, available

oNo extra installs/libraries

Ground-up: hosted at each agencyoPros: data ownership & responsibility, specific needs

oCons: keeping scripts in sync, specific needs

ImplementationSet up environment

Run script tools to create configuration filesoEsri’s Community Addresses Solution

Configure batch file

Run batch fileoManually

oWindows Task Scheduler



replacedata.pybatch.bat runupdate.py

Validate fieldsMap fieldsExtract dataDrop old dataAppend new dataUnique IDsCalc Lat/LongStrip metadata

Schematic

Demo


Lessons learned (so far)Test, test, test: real data & real environments

Customized agency scripts

FunctionsoGenerate a result (e.g. T/F)

oKeep them short & re-useable

ToolsoIDE: PyScripter

oLogging module

oThe internet – document your sources


ProgressInstallations to date

Next stepsoScript integration

oScript data push to Spillman CAD

GIS in the Rockies


Heather Widlund

[email protected]


Special thanks to:West Region Data Integration Group

Questions?

https://github.com/hwidlund/integration.git

automating regional data integration with python & arcpy (heather widlund)

Technology