Data InteroperabilityBest Practices and Common UsesBruce Harold

Dale Lutz

Dale & Bruce

The Basics

What is the Data Interoperability Extension?All About Moving Data From A to B

• What formats?

• From A,B,C… to D,E,F… at once?

• Local data or web too?

• What can I do on the way?

• Is this part of Geoprocessing?

• Can I automate this on a schedule?

• Does it understand OAuth2?

• Do I have to code?

• Can I work with FME?

Where is the Data Interoperability Extension?ArcGIS Desktop, ArcGIS Enterprise

• Desktop means ArcGIS Pro, ArcMap, ArcCatalog…

• Licensed like any other extension

- Named user, concurrent, single user

- Don’t forget to install it

• Analysis Tools includes Quick Tools

• Author Spatial ETL workspaces

- Directly from the ribbon

- Within a toolbox

• Import a Spatial ETL or FME source FMW file

How can I use the Data Interoperability Extension?

• Traditionally three workflows – Direct Read, Quick Conversion, Spatial ETL

• Web GIS and the API economy adds the requirement for Services

Emergent themes: Connectivity, Transformation, Automation

1: Use Workbench 2: It’s Coming!

Capability ArcGIS 10.x ArcGIS Pro 2.2

Direct Read of Formats✓ 1

Quick Format

Conversion ✓ ✓

Spatial ETL Tools✓ ✓

FMW Integration ✓

Publish ETL GP Service✓ 2

How Do We See People Using Data Interoperability?Top Use Cases

• Migration

- Adoption of ArcGIS, Working with ArcGIS

• Connection

- Apps with an API

• Collaboration

- Generate data in a business-friendly format

• Validation

- Feature-Level Business Rule Enforcement

• Detection

- Cross-Format Feature Interaction

• Synchronization

- Feature Services and other repositories

Formats and Repositories

• Formats are data structure

• Web formats are semantic

- JSON

- XML

• Data can be/go anywhere

- Downloaded on demand

- Uploaded to Server, Cloud

- Not tied to ArcGIS Project Catalog

• Increasingly network accessed

• Decreasingly file system

File, Connection, Web – pick any three

APIs too

Direct Format ReadDesktop 10.x Experience

• ArcGIS is Geo-Relational

- Feature classes are database tables with an ObjectID and a geometry column

- Tables have ObjectID but no geometry

• Some file formats can be automatically interpreted this way

- Recognized by file extension (.gml, .geojson, .kml, .gpx, .xlsx, .osm, .xyz….)

- Default behavior assumed

• Many data sources are out of scope for Direct Read

- Not visible in the Catalog

- Interpretation is required

- Custom formats are an option (advanced topic)

• Not feasible in a fully Web GIS

Quick Import and Quick Export

• Easy Button

• When you need data to participate in:

- Editing

- Geodatabase behavior

• Quick Import

- Writes to Geodatabase, Create or Overwrite

- Default Schema

- Postprocessing often required

- In Modelbuilder the Select Data tool accesses feature classes

- Project or Define Projection

• Quick Export

- Writes Any Format (300+ available)

- Default Schema

Multiple FormatsSometimes intermediate data is read or written

• Read Many

• SQL Interact Many

• Spatial Interact Many

• Expression Interact Many

• Write Many

GMLMIFCSVKMLXLSX

Bruce

Best Practices for

Direct Read versus

Quick Tools versus

Spatial ETL Tools

Dale & Bruce

Connect, Transform, Automate

WorkbenchGraphical Programming Application

• Separate Process

• Menus, Controls

• Canvas

- Readers

- Transformers

- Writers

• Windows

- Navigator

- Transformer Gallery

- Log

- Connections

- …

What’s a Transformer?Extract Transform Load

• Transformers are analogous to a geoprocessing tool in ModelBuilder

- Can be grouped, nested, iterated

• Have input and output port(s)

• May read & write filesystem, database, web

• May operate per feature, per group of features or whole dataset

• May operate on attributes and/or geometry

• May operate with parallel concurrency

• Show Me

Bruce

Transformer Basics

Bruce (but I bet Dale chimes in)

Authoring Workspaces

Environment & Options

• FME 2018.0.0.2 Engine

- 300+ Formats

- Feature Caching

- FME Compatible FMW Source

- Database Connections

- Web Connections

- FME Hub Connectivity

• Python Compatibility

- Inherit Pro’s Conda Environment

- Custom Interpreter can be set

Data Interoperability

We could argue all day about how many formats…

What’s FME Hub?In-App Function Download

• Community Platform

- Transformers (in Transformer Gallery)

- Formats

- Templates

- Web Connections

• Search by keyword

• In-App Browsing

- https://hub.safe.com

Custom Transformers

Assemble transformers into logical unit

Declutters the workspace

Looping mechanism

Like grouping in Modelbuilder HTTP Pagination Example

Bruce

Custom

Transformers

Ad Hoc Analysis WorkspacesData Interoperability

• Launch From Ribbon

• Prototype Workflows

- Author

- Open

- Drag In Data or FMW

- Save As Template

- Export As Custom Format

• Send To Email

PythonParameters available as a dictionary

Startup Scripts

Run first

Scripted Parameters

Runtime logic

PythonCreator

Make features

PythonCaller

Operate on features

Shutdown Scripts

Run last

In-App Script Editor

Multiprocessing

Group By Attribute

• Up to 4 processes

WorkspaceRunner

• Up to 7 processes

• Process recycling

Working With PortalsArcGIS Online and ArcGIS Enterprise

• Make Connections to Portals

• Get Background Maps

• Work With Feature Services

- Existing Feature Service

- Create New Feature Service

- Add, Delete Features

- Update Features

- Truncate Services

• Coming in 2019

- ArcGISOnlineConnector

- File upload & download

Integrating ETL Tools and Geoprocessing

• ArcPy access is the same as core geoprocessing

- ImportToolbox

- arcpy.<toolname>_<alias> or arcpy.<alias>.<toolname>

• Wrap ETL tools in ModelBuilder

- Interaction with GP environment and processes

• ETL tool parameters may not match geoprocessing parameter types

- Cast the type in ModelBuilder

A Tool is a Tool

Automation within Geoprocessing

Bruce

Using ETL Tools in

Python &

Modelbuilder

Question Time

• Bruce Harold bharold@esri.com

• Dale Lutz https://www.safe.com/live-chat/

Please Take Our Survey on the App

Download the Esri Events

app and find your event

Select the session

you attended

Scroll down to find the

feedback section

Complete answers

and select “Submit”

See Us Here

• Best Practices and Common Uses

• Best Practices and Common Uses

DATA INTEROPERABILITY WORKSHOP LOCATION

• SDCC 30C

• SDCC 29C

TIME FRAME

• Wed 8:30 – 9:30

• Thur 1:00 – 2:00