Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting Started and Building Workflows Dan Lee Nobbir Ahmed Esri UC 2014 | Technical Workshop | Agenda Geoprocessing Conflation tools: Getting started and Building Workflows What is Conflation? Geoprocessing Conflation Tools Demo 1 Basic scenarios Conflation Workflows Demo 2 Unification of overlapping datasets Demo 3 Edgematching of adjacent datasets Remarks Esri UC 2014 | Technical Workshop | What is Conflation? Geoprocessing Conflation tools: Getting started and Building Workflows Fusione Birletirme Combinao Combinacin Zusammenfhrung Assemblage Translated by Esri localization Esri UC 2014 | Technical Workshop | Challenges arise when multi-source spatial data are used together Common obstacles in analysis and mapping: Spatial and attribute inconsistency caused by differences in data collection and modeling High cost to fix the problems No existing tools can solve the problem effectively Geoprocessing Conflation tools: Getting started and Building Workflows Overlapping datasets Adjacent datasets Esri UC 2014 | Technical Workshop | Conflation reconciles multi-source datasets and optimizes their quality and usability Geoprocessing Conflation tools: Getting started and Building Workflows Conflation is the process of Identifying corresponding features (known as feature matching) Making spatial adjustment and attribute transfer Ultimately, combining matched and unmatched features into one unified dataset with the optimal accuracy, completeness, consistency, and integrity Long-term benefits: No longer living with various imperfect datasets More confidence in reliable analysis and high quality mapping Whats the way to get there? Esri UC 2014 | Technical Workshop | Geoprocessing Conflation Tools Geoprocessing Conflation tools: Getting started and Building Workflows New in ArcGIS Have you used these tools? Esri UC 2014 | Technical Workshop | Our initial focuses Geoprocessing Conflation tools: Getting started and Building Workflows Develop highly automated tools in Geoprocessing framework: Starting with linear features (roads, parcel lines, etc.) Aiming at high feature matching accuracy (not promising 100%) Providing information to facilitate post-processing Build practical workflows Esri UC 2014 | Technical Workshop | Challenges in feature matching (the foundation of conflation) Geoprocessing Conflation tools: Getting started and Building Workflows Dissimilarity Complexity Esri UC 2014 | Technical Workshop | Feature matching (FM) for overlapping datasets Based on proximity, topology, pattern, and similarity analysis, as well as attributes information Geoprocessing Conflation tools: Getting started and Building Workflows 1:1 and 1:m matches m:1 and m:n matches Esri UC 2014 | Technical Workshop | FM-based tool #1 - Detect Feature Changes (DFC) Geoprocessing Conflation tools: Getting started and Building Workflows Update features vs. base features CHANGE_TYPE in output Spatial (S) change Attribute (A) change Spatial and attribute (SA) change No change (NC) New update feature (N) To-Delete base feature (D) DFC Finding feature differences Esri UC 2014 | Technical Workshop | FM-based tool #2 Transfer Attributes (TA) Geoprocessing Conflation tools: Getting started and Building Workflows From source features to target features Transfer fields (e.g. ROAD_NAME) Target features are modified TA Esri UC 2014 | Technical Workshop | FM-based tool #3 Generate Rubbersheet Links (GRL) Geoprocessing Conflation tools: Getting started and Building Workflows Generate Rubbersheet Links (GRL) From source features to target features Followed by Rubbersheet Features (RF) Adjusting input features GRLRF Rubbersheeting moves source locations towards target locations based on established links Esri UC 2014 | Technical Workshop | Edge (feature) matching (EM) and tools for adjacent datasets Based on proximity, topology, and continuity analysis, as well as attributes information Geoprocessing Conflation tools: Getting started and Building Workflows Generate Edgematch Links (GEL) From source features to adjacent features Followed by Edgematch Features (EF) Connects features guided by the established links GEL EF Esri UC 2014 | Technical Workshop | Demo 1: Basic scenarios Geoprocessing Conflation tools: Getting started and Building Workflows Part 2: Edgematch adjacent datasets Part 1: Conflate overlapping datasets Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows A popular scenario and requirements: To unify the two datasets into one with combined spatial and attribute information Part 1: conflation of overlapping datasets setA Contains updates setA Contains updates setB Spatially accurate setB Spatially accurate setC Best of both setC Best of both (1) Make a copy (4) For matched: Transfer uncommon attributes (3) For all features: Do rubbersheeting adjustment (6) Append unmatched (2) Identify matched and unmatched (5) Select unmatched Workflow strategy Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows Part 1: Input streets Base features with spatial accuracy and attributes Update features with new streets and attributes Together Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows This reflects the conflation strategy. With the simple and highly similar demo data, the process produces 100% accurate result. Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows Part 1: results Attributes transferred Changes detected Rubbersheeting links generated New features adjusted and added to base Esri UC 2014 | Technical Workshop | Part 2: edgematching of adjacent datasets Goal - make two adjacent datasets connect Geoprocessing Conflation tools: Getting started and Building Workflows Source features Adjacent features Together Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows Since the demo data has little ambiguity; the process produces 100% accurate results. Esri UC 2014 | Technical Workshop | Part 2: results Geoprocessing Conflation tools: Getting started and Building Workflows Move endpoint Esri UC 2014 | Technical Workshop | Conflation Workflows Geoprocessing Conflation tools: Getting started and Building Workflows Unification of overlapping datasets Edgematching of adjacent datasets Esri UC 2014 | Technical Workshop | Three components in conflation workflows In same projection Data validation Selection of relevant features Conflation tools Workflow tools Queued review Interactive editing Geoprocessing Conflation tools: Getting started and Building Workflows Preprocessing Conflation and evaluation Conflation and evaluation Postprocessing Esri UC 2014 | Technical Workshop | Demo 2: Unification of overlapping datasets Geoprocessing Conflation tools: Getting started and Building Workflows Workflow strategy Esri UC 2014 | Technical Workshop | Overview Geoprocessing Conflation tools: Getting started and Building Workflows Two road datasets (northeast of Meigs County, OH): LocalNE 1085 features StateNE 1013 features Both datasets: Have common and uncommon features and attributes Are well preprocessed Esri UC 2014 | Technical Workshop | Breakdown of the workflow ( based on Demo 1 Part 1 workflow ) Geoprocessing Conflation tools: Getting started and Building Workflows QA #1 QA #2 QA #4 QA #3 Step1aStep1a Step4Step4 Step2Step2 Step3Step3 Step5Step5 Step1bStep1b Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows Step1a of the workflow with evaluation QA #1 Esri UC 2014 | Technical Workshop | DFC result and potential match errors Geoprocessing Conflation tools: Getting started and Building Workflows Esri UC 2014 | Technical Workshop | QA potential match errors Total 16 CFM_GRP were flagged; 11 had match issues due to data complexity and dissimilarity; 5 were ignorable Geoprocessing Conflation tools: Getting started and Building Workflows Match issue due to data complexity Match issue ignorable Esri UC 2014 | Technical Workshop | QA DFC result CHANGE_TYPE D and N ((CHANGE_TYPE = 'N') OR ( CHANGE_TYPE = 'D' )) AND( (NEAR_DIST > 0) AND (NEAR_DIST < 10)) Geoprocessing Conflation tools: Getting started and Building Workflows Inspect records with high potential for errors: 35 reviewed 11 wrong Ns or Ds flagged Wrong N Wrong D Esri UC 2014 | Technical Workshop | Matched groups: Total: 896 groups Correct: 885 groups Incorrect: 11 groups Accuracy = 885 / 896 = 98.77% Unmatched: Total: 240 (155 Ns + 85 Ds) Correct: 229 (151 Ns + 78 Ds) Incorrect: 11 (4 Ns + 7 Ds) Accuracy = 229 / 240 = 95.42% (biased by the total count) Ready to join with inputs to tag Ns and Ds Feature matching accuracy estimates Overall feature matching accuracy (average of matched and unmatched) % Geoprocessing Conflation tools: Getting started and Building Workflows Esri UC 2014 | Technical Workshop | Extract Matched Features as inputs for GRL LocalNE: 934 non-N out of 1085 StateNE: 935 non-D out of 1013 Ready for GRL process Step1b of the workflow Geoprocessing Conflation tools: Getting started and Building Workflows Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows Step2 of the workflow with evaluation and QA QA #2 Esri UC 2014 | Technical Workshop | GRL result Geoprocessing Conflation tools: Getting started and Building Workflows Generated total regular links and identity links Esri UC 2014 | Technical Workshop | GRL evaluation results Intersecting links Geoprocessing Conflation tools: Getting started and Building Workflows 54 locations of intersecting links; Esri UC 2014 | Technical Workshop | GRL evaluation results links of different src/tgt vertex types Geoprocessing Conflation tools: Getting started and Building Workflows (qaNotes = 'src_tgt_VxType_diff') AND( (tgtVxType >=2) OR( srcVxType >=2 )) AND NEAR_DIST = of flagged links were more important Esri UC 2014 | Technical Workshop | GRL evaluation results locations of missing links Geoprocessing Conflation tools: Getting started and Building Workflows 22 of the 595 source locations of missing links were on nodes; all others are on in-line vertices. 20 ORIG_FID of frequency >5 locations were reviewed and confirmed non-critical. Esri UC 2014 | Technical Workshop | QA regular links - summary Geoprocessing Conflation tools: Getting started and Building Workflows Total 241 (0.92%) of links were reviewed: 44 were modified 86 were to be removed 111 were ok 42 missing link locations were reviewed: 14 links were added Links at other locations were not critical Ready for rubbersheeting (qaNotes = 'src_tgt_VxType_diff') AND( (tgtVxType >=2) OR( srcVxType >=2 )) Esri UC 2014 | Technical Workshop | regular links were selected by (REV_FLAG 'Delete') OR( REV_FLAG IS NULL) to participate Geoprocessing Conflation tools: Getting started and Building Workflows Step3 of the workflow with assessments QA #3 Esri UC 2014 | Technical Workshop | Rubbersheeting result Geoprocessing Conflation tools: Getting started and Building Workflows Esri UC 2014 | Technical Workshop | GRL result after rubbersheeting Many regular links became identify links Geoprocessing Conflation tools: Getting started and Building Workflows Esri UC 2014 | Technical Workshop | How good is the rubbersheeting result? Three indicators showing spatial improvement Geoprocessing Conflation tools: Getting started and Building Workflows Less spatial differences Before RFAfter RF Regular links Identity links Improved location alignment Link-length distributions before/after RF - spatially closer to target (Not on the same scale due to the big difference in values) Esri UC 2014 | Technical Workshop | QA #3 Check rubbersheeting result Source (original) and target Target features Source adjusted with N features highlighted Ready to do TA Geoprocessing Conflation tools: Getting started and Building Workflows Esri UC 2014 | Technical Workshop | Transfer attributes (UC2014_ID) from adjusted source to target Excluding Ns from adjusted source; excluding Ds from joined target Geoprocessing Conflation tools: Getting started and Building Workflows Step4 of the workflow with evaluation and QA QA #4 Esri UC 2014 | Technical Workshop | Attribute transfer result Geoprocessing Conflation tools: Getting started and Building Workflows Esri UC 2014 | Technical Workshop | QA #4 Check attribute transfer result NEAR_DIST >=0; no-transfer features found nearby source features for potentially missed matches 32 records were reviewed: 18 were edited with UC2014_ID values Almost there Geoprocessing Conflation tools: Getting started and Building Workflows Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows Final step of the full workflow (CHANGE_TYPE = 'N') AND(( REV_FLAG 'wrongN' ) OR REV_FLAG IS NULL) Select N features from adjusted source and append them to target Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows Appended N features in final result Esri UC 2014 | Technical Workshop | Unification of overlapping datasets workflow completed! Processing Time Step 1 (a, b)1 min 3 sec Step 21 min 14 sec Step 31 min Steps 4, 518 sec Total3 min 35 sec Automated processing Interactive processing (not counting final review) QA #1 (CFM_GR P and DN) QA #2 (links) QA #3 QA #4 (attribute transfer) QA Total Time (2-3 review counts per minute) Review Count (locations or feature groups) 51283x32366 ~ 2-3 hrs. Edit Count (field values) 46255x18319 Geoprocessing Conflation tools: Getting started and Building Workflows Esri UC 2014 | Technical Workshop | Demo 3: Edgematching of adjacent datasets Geoprocessing Conflation tools: Getting started and Building Workflows Workflow strategy Esri UC 2014 | Technical Workshop | Breakdown of the workflow ( based on Demo 1 Part 2 ) Geoprocessing Conflation tools: Getting started and Building Workflows Step6aStep6a QA #1 Step7Step7 Step6bStep6b Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows Overview Two road datasets (an area in Alabama): EdgeRoads 7576 features GISRoads 3634 features Both datasets: Contain roads that are within 1 km to borders Have inconsistent road names Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows Step6a of the workflow QA #1 Esri UC 2014 | Technical Workshop | GEL result Geoprocessing Conflation tools: Getting started and Building Workflows Generated 454 links; midpoints of links were created for visualization purpose. Borders were not in the process, but displayed for reference. Esri UC 2014 | Technical Workshop | Example edgematch links Geoprocessing Conflation tools: Getting started and Building Workflows EM_CONF in output 100 (matched with no ambiguity) 50 (spatially matched with unmatched attributes) < 50 (spatially matched with some ambiguity and weak continuity) Esri UC 2014 | Technical Workshop | GEL evaluation results Geoprocessing Conflation tools: Getting started and Building Workflows 33 locations of intersecting links; 134 links of low EM_CONF values; 62 source dangle locations of potential missing links Esri UC 2014 | Technical Workshop | Inspection and editing of edgematch links Geoprocessing Conflation tools: Getting started and Building Workflows Reviewed: 33 locations of intersecting links NEAR_DIST >= 0 98 low EM_CONF links (EM_CONF < 33) AND (REV_FLAG IS NULL) 62 source dangle locations (near links) Summary: 388 (~85%) of total 459 links were good (54 were flagged for recheck) 71 (~15%) of total links were modified, removed, or added Esri UC 2014 | Technical Workshop | What happened to the SRC_FID and TGT_FID of the added or modified links? Geoprocessing Conflation tools: Getting started and Building Workflows Ready for edgematching Step6b of the workflow Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows Step7 of the workflow Esri UC 2014 | Technical Workshop | Edgematch result Geoprocessing Conflation tools: Getting started and Building Workflows Review flagged locations Esri UC 2014 | Technical Workshop | Edgematching of adjacent datasets workflow completed! Processing Time Step 6a6.52 sec Step 6b4.09 sec Step 6c2.15 sec Total12.76 sec Automated processing Interactive processing (not counting final review) QA Links Time (2-3 review counts per minute) Review Count (locations or feature groups) ~ 193 ~ hrs. Edit Count (field values) 192 Geoprocessing Conflation tools: Getting started and Building Workflows Esri UC 2014 | Technical Workshop | Remarks Geoprocessing Conflation tools: Getting started and Building Workflows Thanks to: Department of Public Works (DPW), Los Angeles County, USA. Institut Cartogrfic i Geolgic de Catalunya (ICGC), Barcelona, Spain. Ohio State Department of Transportation, USA. National Institute for Water and Atmospheric Research (NIWA) and Land Information New Zealand (LINZ) - Crown Copyright Reserved. All others who supported us along the way. Esri UC 2014 | Technical Workshop | Conflation can be done more efficiently now in ArcGIS Desktop Geoprocessing Conflation tools: Getting started and Building Workflows Use the best practice Highly accurate results and rich information Small among of review and editing Time worth-spending Preprocessing Conflation and evaluation Conflation and evaluation Postprocessing Esri UC 2014 | Technical Workshop | Our future work Geoprocessing Conflation tools: Getting started and Building Workflows New tools and enhancements Transform Features, Align Features Other feature types Contextual conflation (spatially related features) Integrated review-editing environment Formalization of workflows Common scenario-oriented Guidelines and training Please send us your use cases and requirements Esri UC 2014 | Technical Workshop | We recommend you to consider conflation a higher priority Geoprocessing Conflation tools: Getting started and Building Workflows Study the tools and understand the results Use small test areas Begin to build workflows for your organizations Improve data quality and usability Work with broader communities Community mapping Please send us your feedbacks Esri UC 2014 | Technical Workshop | Recent papers and other references Geoprocessing Conflation tools: Getting started and Building Workflows Lee, D., Yang, W., Ahmed, N. (2014). Conflation in Geoprocessing Framework - Case Studies, GEOProcessing, 2014, Barcelona, Spain.Yang, W., Lee, D., Ahmed, N. (2014). Pattern Based Feature Matching for Geospatial Data Conflation, GEOProcessing, 2014, Barcelona, Spain.V. Walter and D. Fritsch, Matching spatial data sets: a statistical approach, International Journal of Geographical Information and science, vol. 3(5, 1999), pp M. Zhang and L. Meng, Delimited stroke oriented algorithm working principle and implementation for the matching of road networks, Journal of Geographic Information Sciences, vol. 14(1), June, 2008, pp L. Li and M. Goodchild, An optimization model for linear feature matching in geographical data conflation, International Journal of Image and Data Fusion, vol. 2(4), 2011, pp Esri UC 2014 | Technical Workshop | Please fill out the session survey: First Offering ID: 1116 Second Offering ID: 1227 Online Paper pick up and put in drop box Geoprocessing Conflation tools: Getting started and Building Workflows Reminder Thank you for attending! Any questions, comments ? Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows