arcgis data reviewer: an introduction · workshop scenario my organization needs to improve data...
TRANSCRIPT
ArcGIS Data Reviewer: An IntroductionJay Cary, Kumar Gaurav
Workshop agenda
• Defining data quality
• What is ArcGIS Data Reviewer?
• Automated review
• Semi-automated review
• Error management and reporting
• Summary/resources
ArcGIS Data Reviewer: An Introduction
Defining data quality
ArcGIS Data Reviewer: An Introduction
Defining qualityA technical perspective
Spatial Accuracy
ISO-19157:2013 Geographic information – Data quality
ArcGIS Data Reviewer: An Introduction
Lake feature has been shifted
Defining qualityA technical perspective
Spatial Accuracy
Thematic Accuracy
ISO-19157:2013 Geographic information – Data quality
ArcGIS Data Reviewer: An Introduction
Swimming pool captured as wetland
Defining qualityA technical perspective
Spatial Accuracy
Thematic Accuracy
Completeness
ISO-19157:2013 Geographic information – Data quality
ArcGIS Data Reviewer: An Introduction
Neighborhood with missing building footprint
Defining qualityA technical perspective
Spatial Accuracy
Thematic Accuracy
Completeness
Logical Consistency
ISO-19157:2013 Geographic information – Data quality
ArcGIS Data Reviewer: An Introduction
Highway with road surface type gravel
Defining qualityA technical perspective
Spatial Accuracy
Thematic Accuracy
Completeness
Logical Consistency
Temporal Quality
ISO-19157:2013 Geographic information – Data quality
ArcGIS Data Reviewer: An Introduction
Updated chart should show closed runway
Outdated chart showing open runway
Defining qualityA technical perspective
Spatial Accuracy
Thematic Accuracy
Completeness
Logical Consistency
Temporal Quality
Usability
ISO-19157:2013 Geographic information – Data quality
ArcGIS Data Reviewer: An Introduction
Used to map National Parks
Used to route emergency vehicles
Defining qualitySources of data quality requirements
ArcGIS Data Reviewer: An Introduction
Industry
standards /
specifications
Defining qualitySources of data quality requirements
ArcGIS Data Reviewer: An Introduction
Subject matter
experts
Industry
standards /
specifications
Defining qualitySources of data quality requirements
ArcGIS Data Reviewer: An Introduction
Subject matter
experts
Industry
standards /
specifications
Training and
experience
Defining qualitySources of data quality requirements
ArcGIS Data Reviewer: An Introduction
Subject matter
experts
Industry
standards /
specifications
Quality
assurance plan
Training and
experience
Data quality managementCapabilities of ArcGIS
ArcGIS Data Reviewer: An Introduction
Geodatabase integrity
• Schema constraints
• Attribute rules
• Versioning
Advanced data types
• Topologies
• Parcel fabric
• Utility network
ArcGIS
Data Reviewer
• Automated review
• Semi-automated review
• Error management
• Quality reporting
What is ArcGIS Data Reviewer?Data quality management in ArcGIS
• ArcGIS Desktop
- ArcMap extension
- ArcGIS Pro extension
• ArcGIS Server
• Data Reviewer API
- JavaScript
- ArcGIS Pro SDK for Microsoft .NET
• Web AppBuilder for ArcGIS
ArcGIS Data Reviewer: An Introduction
Apps
Desktop
APIs
Enterprise
Managing quality controlQuality control processes
ArcGIS Data Reviewer: An Introduction
Automated
review
Quality
reporting
Semi-automated
review
Implementing
automated review
ArcGIS Data Reviewer: An Introduction
Managing quality controlQuality control processes
Automated
review
Quality
reporting
Semi-automated
review
ArcGIS Data Reviewer: An Introduction
Types of Quality Control
Fast
Consistent and
repeatable
Objective
100% coverage
Automated
review
ArcGIS Data Reviewer: An Introduction
After data creation
During data creation
Automating data review
• Implementing quality requirements
- 40+ configurable checks
- Feature integrity
- Collection rules
ArcGIS Data Reviewer: An Introduction
Cutbacks Duplicate Vertex
Evaluate Part Count
Invalid Geometry
Monotonicity Polyline/Path Closes Self
Automating data review
• Implementing quality requirements
- 40+ configurable checks
- Feature integrity
- Collection rules
- Attribution
- Feature and table values
ArcGIS Data Reviewer: An Introduction
Domain Query Attributes
Regular Expression
Subtypes Table to Table Attribute
Unique ID
Automating data review
• Implementing quality requirements
- 40+ configurable checks
- Feature integrity
- Collection rules
- Attribution
- Feature and table values
- Spatial
- Spatial relationships
ArcGIS Data Reviewer: An Introduction
Duplicate Feature Event on Event Feature on Feature
Find Dangles Polygon Overlap/Gap is
Sliver
Unnecessary Polygon Boundaries
Automating data review
• Implementing quality requirements
- 40+ configurable checks
- Feature integrity
- Collection rules
- Attribution
- Feature and table values
- Spatial
- Spatial relationships
ArcGIS Data Reviewer: An Introduction
http://esriurl.com/12379
Implementing data quality rulesReviewer Rule implementation workflow
Authoring Reviewer Rules
- Identify data quality requirements
- What ArcGIS capability supports the requirement?
- What workflow will the capability support?
Industry
standards /
specifications
Subject matter
experts
Training and
experience
Quality
assurance
plans
ArcGIS Data Reviewer: An Introduction
Validation
Constraint
Implementing data quality rulesReviewer Rules in constraint workflows
• Attribute (constraint) rules
- Created using ArcGIS Pro
- Validates data sources
- Geodatabase features/tables
- 9 check types
- Rules stored in Geodatabase
- Supported in
- File Geodatabase
- Enterprise Geodatabase
- Feature Services
ArcGIS Data Reviewer: An Introduction
Implementing data quality rulesReviewer Rules in validation workflows
• Map rules
- Created using ArcGIS Pro
- Validates layers
- Geodatabase features/tables
- Shapefiles
- Feature services
- 10 check types
- Rules stored in Map
- Supported in
- ArcGIS Pro
ArcGIS Data Reviewer: An Introduction
Implementing data quality rulesReviewer Rules in validation workflows
• Reviewer Batch Jobs
- Created using ArcMap
- Validates data sources
- Geodatabase features/tables
- Shapefiles
- 43 check types
- Rules stored in a file
- Supported in
- ArcGIS Desktop
- ArcGIS Pro
- ArcGIS Server
ArcGIS Data Reviewer: An Introduction
Demonstration
Authoring
data quality rules
ArcGIS Data Reviewer: An Introduction
Workshop scenario
My organization needs to improve data quality to support future automation of task for finding
and routing to locations.
Success criteria
❑ Future data quality requirements are integrated into existing data management workflows
❑ Existing and new features should comply with new quality requirements
❑ Features should be accurately positioned
❑ All errors are corrected and verified
ArcGIS Data Reviewer: An Introduction
Demo Scenario
Industry
standards /
specifications
Subject
matter
experts
Training and
experience
Quality
assurance
plans
Validation
Constraint
ArcGIS Data Reviewer: An Introduction
Questions?
ArcGIS Data Reviewer: An Introduction
Automated review
ArcGIS Data Reviewer: An Introduction
Methods for executing automated reviewValidating features during editing using Reviewer Rules
Constraint workflows
• Triggered during creation,
modification or deletion of a
feature
• Error message displayed when
feature violates constraint
ArcGIS Data Reviewer: An Introduction
Methods for executing automated reviewValidating features using map-based Reviewer Rules
Validation workflows
• ArcGIS Pro
• Validates
- Selection
- Map extent
- Changed features (versioning)
ArcGIS Data Reviewer: An Introduction
Methods for executing automated reviewValidating features using Reviewer Batch Jobs
Validation workflows
• ArcMap
ArcGIS Data Reviewer: An Introduction
Methods for executing automated reviewValidating features using Reviewer Batch Jobs
Validation workflows
• ArcMap
• Geoprocessing
ArcGIS Data Reviewer: An Introduction
Methods for executing automated reviewValidating features using Reviewer Batch Jobs
Validation workflows
• ArcMap
• Geoprocessing
• ArcGIS Workflow Manager
ArcGIS Data Reviewer: An Introduction
Methods for executing automated reviewValidating features using Reviewer Batch Jobs
Validation workflows
• ArcMap
• Geoprocessing
• ArcGIS Workflow Manager
• ArcGIS Server
ArcGIS Data Reviewer: An Introduction
http://bit.ly/DR_BVM
Methods for executing automated reviewValidating features using Reviewer Batch Jobs
Validation workflows
• ArcMap
• Geoprocessing
• ArcGIS Workflow Manager
• ArcGIS Server
• Validates
- Selection
- Map extent
- Full database
- Changed features (traditional versioning)
ArcGIS Data Reviewer: An Introduction
http://bit.ly/DR_BVM
Demonstration
Automated review
ArcGIS Data Reviewer: An Introduction
Workshop scenario
My organization needs to improve data quality to support future automation of task for finding
and routing to locations.
Success criteria
✓ Future data quality requirements are integrated into existing data management workflows
❑ Existing and new features should comply with new quality requirements
❑ Features should be accurately positioned
❑ All errors are corrected and verified
ArcGIS Data Reviewer: An Introduction
Questions?
ArcGIS Data Reviewer: An Introduction
Semi-automated review
ArcGIS Data Reviewer: An Introduction
Managing quality controlQuality control processes
Automated
review
Quality
reporting
Semi-automated
review
ArcGIS Data Reviewer: An Introduction
Types of quality control
Fast
Consistent and
repeatable
Objective
100% coverage
Automated
review
Semi-automated
review
Guided Workflows
Streamlined processes
Subjective
Sampling
ArcGIS Data Reviewer: An Introduction
Value of performing visual review
Discover patterns
Compare against trusted sources
Find missing features
ArcGIS Data Reviewer: An Introduction
Semi-automated reviewLeveraging ArcGIS Desktop and Server
Tools supporting semi-automated review
- Redlining missing features
- Flagging existing features in error
- Random sampling
ArcGIS Data Reviewer: An Introduction
Demonstration
Visual data review
ArcGIS Data Reviewer: An Introduction
Workshop scenario
My organization needs to improve data quality to support future automation of task for finding
and routing to locations.
Success criteria
✓ Future data quality requirements are integrated into existing data management workflows
✓ Existing and new features should comply with new quality requirements
❑ Features should be accurately positioned
❑ All errors are corrected and verified
ArcGIS Data Reviewer: An Introduction
Questions?
ArcGIS Data Reviewer: An Introduction
Error management
and reporting
ArcGIS Data Reviewer: An Introduction
Reporting data quality
What are the issues on a geographic extent?
How severe are the issues with my data?
How many types of issues do I have in my data?
How many issues have been reviewed and fixed?
How many issues do I have by source?
ArcGIS Data Reviewer: An Introduction
Reviewer Errors
• Spatial and non-spatial information
• Attributes describe check, problem, and validation information
• Identifier provides link back to feature that failed validation
• Have a defined lifecycle (Reviewed, Corrected, Verified)
Reviewer
Errors
ArcGIS Data Reviewer: An Introduction
Managing quality controlQC lifecycle management
ArcGIS Data Reviewer: An Introduction
Review
Find and
record errors
Correct
Perform editsor
Note exceptions
Verify
Acceptableor
Unacceptable
Demonstration
Managing Errors
ArcGIS Data Reviewer: An Introduction
Workshop scenario
My organization needs to improve data quality to support future automation of task for finding
and routing to locations.
Success criteria
✓ Future data quality requirements are integrated into existing data management workflows
✓ Existing and new features should comply with new quality requirements
✓ Features should be accurately positioned
❑ All errors are corrected and verified
ArcGIS Data Reviewer: An Introduction
Questions?
ArcGIS Data Reviewer: An Introduction
Workshop review
• Defining data quality
• Forms of data quality control
- Automated review
- Semi-automated review
• ArcGIS Data Reviewer
- Automated validation checks
- Semi-automated tools
- Error management
ArcGIS Data Reviewer: An Introduction
See Us Here
Tuesday
ArcGIS Data Reviewer: An Introduction (Offering #1) Room 30 C 08:30 am - 09:30 am
Wednesday
ArcGIS Data Reviewer: An Introduction (Offering #2) Room 30 E 08:30 am - 09:30 am
ArcGIS Data Reviewer: Implementing Ready-to-Use Rules across the
platform
Demo Theater 4 10:00 am - 10:45 am
ArcGIS Data Reviewer: Deploying Data Quality Web Services Demo Theater 5 11:15 am - 12:00 pm
ArcGIS Data Reviewer: Validating linear-referenced events Demo Theater 4 11:15 am - 12:00 pm
Thursday
ArcGIS Data Reviewer: Advanced Data Validation Room 30 E 08:30 am - 09:30 am
ArcGIS Data Reviewer: An Introduction
See Us Here
ArcGIS Data Reviewer: An Introduction
Data Reviewer
Info
Want to learn more?
• Product page
www.esri.com/datareviewer
ArcGIS Data Reviewer: An Introduction
Want to learn more?
• Data Reviewer templates
- Local government
- Topographic mapping
- Utilities
- Electric
- Gas
- Water
- Water resources
• Based on Esri industry models
• Use as starting point
ArcGIS Data Reviewer: An Introduction
solutions.arcgis.com
Want to learn more?
• Documentation
- Desktop desktop.arcgis.com
ArcGIS Data Reviewer: An Introduction
Want to learn more?
• Documentation
- Desktop
- Server
enterprise.arcgis.com
ArcGIS Data Reviewer: An Introduction
Want to learn more?
• Documentation
- Desktop
- Server
• Training
http://esriurl.com/15503
ArcGIS Data Reviewer: An Introduction
Want to learn more?
• Documentation
- Desktop
- Server
• Training
• GeoNet Community
http://esriurl.com/15504
ArcGIS Data Reviewer: An Introduction
Please Share Your Feedback in the App
Download the Esri
Events app and find
your event
Select the session
you attended
Scroll down to
“Survey”
Log in to access the
survey
Complete the survey
and select “Submit”
ArcGIS Data Reviewer: An Introduction
Questions?
ArcGIS Data Reviewer: An Introduction