Data Interoperability

An IntroductionBruce Harold

Dale Lutz

Workshop TopicsArcGIS Data Interoperability

• Product Value Proposition

- Interoperability Scope

• Basic Format Import & Export

- Demo: Quick Tools

• Visual ETL Programming

- Demo: The Workbench Application

• Overcoming Interoperability Challenges

- Demo: Harvest Spatial CSV from CKAN

• Synchronizations

- Demo: GeoServer to ArcGIS Online Mirroring

• Your Future

- Demo: Web App Integration

What Does ArcGIS Data Interoperability Do?Integrates apps A and B with data, optionally transformed

• What kind of apps can A & B be?

• From A,B,C… to D,E,F… at once?

• Local data or web too?

• What can I do on the way?

• Is this part of Geoprocessing?

• Can I automate this on a schedule?

• Does it understand authentication?

• Can I work with FME?

• Do I have to write code?

That Was Then, This Is Now

• Began with format support

- Esri leveraged Safe’s best of breed spatial format support

• Direct read was important

- CAD, Geomedia Warehouse, KML, GML, XML

• Important file types are in core ArcGIS now

- DWG, DGN, Revit, KML, GeoJSON

• Sources have gone to the web

• Data has modes

- System of Record (e.g. RDBMS), App Specific (e.g. XLS), Message (e.g. EsriJSON)

• Important format is now semantic and can be used in any mode

- JSON delivered via REST

Interoperability means just that, it moves with the times

Did you know a URL can be a file path? Did you know you can handle web service calls without coding?

Sources and RepositoriesFile, Connection, Web – pick any three

• Pick what you want

• Search experience

• Parameters vary

• Extensible with Custom Formats

FME Environment Highlights

• FME 2019.0.0.0 Engine, FMW Compatibility

- 370+ Formats Data Sources

- “Favorites” may be persisted

- Database & Web Connections, Python Environment

- FME Hub Connectivity

• New Capabilities

- FME Packages (no creeping technical debt)

- Raster Object Detection

- Natural Language Processing

- Change Detector “reporting” behavior

- Esri Projection Engine embedded

Data Interoperability in Pro 2.4

We could argue all day about how many formats data sources…

Where is the Data Interoperability Extension?ArcGIS Desktop, ArcGIS Enterprise

• Its an Extension

- Install it

- License it

• Data Interoperability System Tools

- Quick Import, Quick Export

• Create Spatial ETL Tools in Project Toolboxes

• Start Workbench app from the Analysis ribbon

• ETL Tool source can be a FMW file

- Share with FME users

“Quick Tools” TranslationsImport to File Geodatabase, Export from Geodatabase

• Organizations tend to pick a favorite publication format

• Audience may be mixed, non-technical and technical

• Many ‘interchange’ formats are natively supported in core ArcGIS Pro

- CSV, KML, GeoJSON, Excel…

• Others ‘need work’ to get right

- GML, XML…

• Quick Import robustly delivers a default translation to Geodatabase

• Quick Export goes the opposite direction

• Demo will showcase OGR VRT

VRT to GDB

GDB to GPKG

Bruce

Quick Tools Demo

Extract and Load I Understand, What Is Transform?Extract Transform Load

• Functions applied to your data

• Configured, not Coded

• Hundreds of Transformers ship out of the box

• Organized in Categories

• In-App Downloadable from FME Hub

• Operate per feature or per dataset

• Connect in streams by diagramming

Transformers Have Meaningful Names

• Operations on…

- Geometry

- Attribute schema

- Attribute values

- Properties (styling)

• Route features

• Filter features

• Python is available

Web Source

Schema Manipulation

Bruce

Simple Powerful

ETL in Workbench

Solving Interoperability ChallengesPowerfully, without coding

• The data is somewhere else

• You don’t control it

• The format is simple, maybe too simple

- CSV, Excel

• It has quality issues

- Spatial data is in various attributes

- Some coordinates are outliers

- Attribute names, casing vary

• No Problem!

3rd Party Portal

Variable Location Attributes

Incomplete Geocode Values

Bruce

Overcoming

Interoperability

Challenges

Synchronization: Major Use Case

• Non-Esri system of record

• Data has gravity, may stay where it is

• Portals like CKAN, DKAN…

• Non-Esri OGC services

- WFS

• Atom Feeds

• Publish Data to ArcGIS Online/Enterprise

- Performs in ArcGIS Apps

GeoServer Source

Bruce

Efficient WFS

Synchronization to

ArcGIS Online

Join The API Economy

• Workbench & REST democratizes integration

• Learn a few simple techniques

- Authentication methods

- Constructing a URL

- HTTPCaller transformer and GET versus POST

- Query parameters & headers

- JSON handling

- Pagination

• You can conquer code-free integrations!

Data Interoperability; Integration in the Modern Commercial Enterprise

Customer

Support

Analytics

& BI

CRM

Design

Business

Process

Automation

+

ERP

Datastores

Cloud File SystemsEnsuring ArcGIS Presence in Low-Code App Environments

Integration

Services

ArcGIS

Marketing &

Social

Business Search

ArcGIS Online Destination

Bruce

Web App

Integration

Where To Learn More

• Online Help

• GeoNet Space

• FME Documentation

• FME Hub

• Safe KnowledgeBase

• Free Evaluation

ArcGIS Data Interoperability

Questions

Please Share Your Feedback in the App

Download the Esri

Events app and find

your event

Select the session

you attended

Scroll down to

“Survey”

Log in to access the

survey

Complete the survey

and select “Submit”