arcgis geoevent server: a developer's guide€¦ · arcgis geoevent server: a developer's...

ArcGIS GeoEvent Server:

A Developer's GuideMark Bramer

Esri Professional Services

Vienna, VA

Agenda

• Connectors

- Inbound

- Outbound

• GeoEvent SDK

• Transports and Adapters

• Processors

• Maven

• Scripting “upstream”

Real-Time GISIntegration and exploitation of streaming data

• Integrates real-time

streaming data

into ArcGIS

• Performs continuous

processing and

real-time analytics

• Sends updates and alerts

to those who need it

where they need it

ArcGIS Enterprise

GeoEvent

Server

DesktopWeb Device

DesktopWeb Device

live & historic

aggregates & features

map & feature service

• Ingest high velocity real-time IoT

data into ArcGIS.

• Perform continuous analytics on

IoT events as they are received.

• Store IoT observations in a

spatiotemporal big data store.

• Visualize high velocity &

volume IoT data:

- as an aggregation

- or as discrete features.

• Notify about IoT patterns of

interest.

• Adjust behavior of things in our

environment through actuation.

stream service

live features

ArcGIS

Enterprise

GeoEvent

Server

ingestion

analytics

ArcGIS as an IoT PlatformArcGIS Enterprise with real-time & big data capabilities

GeoAnalytics

Server

spatiotemporal

big data store

storage analytics

visualization

actuation

Connectors

ConnectorsWhat is a connector?

• Connectors get event data into, and out of, GeoEvent Server

• Every connector has two components

- Transport

- Adapter

• Transports and Adapters can be inbound or outbound:

- Inbound transport – connects to and gets raw bytestream

- Inbound adapter – converts raw bytestream into a geoevent

- Outbound adapter – converts geoevent into a byte array, formatted for an output type

- Outbound transport – accepts arrays of bytes from the adapter and transmits them

ConnectorsConfiguring a Connector

• The connector helps the user by

▪ Providing default values

▪ Re-label properties to be appropriate to the context

▪ Move properties under an “advanced” area to discourage modification

▪ Completely hide properties that the user should not see

Receiving Real-Time DataEasily integrate real-time streaming data into ArcGIS using an Input Connector

You can create

your own

connectors.

GeoEvent Server

Inp

uts

Ou

tpu

ts

GeoEvent Services

Poll an ArcGIS Server for Features

Poll an external website for GeoJSON, JSON, or XML

Receive Features, GeoJSON, JSON, or XML on a REST endpoint

Receive RSS

Receive GeoJSON or JSON on a WebSocket

Receive Text from a TCP or UDP Socket

Subscribe to an external WebSocket for GeoJSON or JSON

Watch a Folder for new CSV, GeoJSON, or JSON Files

Ou

t o

f th

e B

ox

REST

.csv

WS

WS

HTTP

Esri

Gallery

ActiveMQ

CAP

Instagram

Exploitation Support Data

Cursor-on-Target

RabbitMQ

NMEA 0183

MQTT

Sierra Wireless (RAP)

KML

Kafka

Trimble (TAIP)

Twitter

Part

ner

Gallery

CompassLDE

enviroCar

GNIP

FAA (ASDI)

exactEarth AIS

Zonar

Valarm

Networkfleet

OSIsoft *

*

*

*

*

you can create

your own

connectors

GeoEvent Server

Inp

uts

Ou

tpu

ts

GeoEvent Services

Feature-JSON

Connectors Transport Adapter

GeoJSON

Receive Feature-JSON from Kafka

Receive GeoJSON on a REST endpoint

Feature-JSON

Adapters

GeoJSON

JSON

RSS

Text

XML

CAP

Cursor-on-Target

GeoMessage

Instagram

NMEA

Sierra Wireless (RAP)

Trimble (TAIP)

Twitter

VMF

Es

riG

all

ery

Waze

Feature Service

Transports

File

HTTP

HTTP+BasicAuth

HTTP+OAuth

TCP

UDP

Waze

WebSocket

Esri

Gallery ActiveMQ

IRC

Kafka

MQTT

RabbitMQ

Twitter

Kafka

HTTP

Receiving Real-Time DataInput Connector = Transport + Adapter

You can create

your own

connectors.

Sending Real-Time DataEasily disseminate notifications, alerts, and updates using an Output Connector

GeoEvent Server

Inp

uts

Ou

tpu

ts

GeoEvent Services

Ou

t o

f th

e B

ox

Add or Update a feature

Publish Text to a UDP Socket

Send a Text Message

Send an Email

Push Text to an external TCP Socket

Push GeoJSON or JSON to an external WebSocket

Push GeoJSON or JSON to an external Website

Send an Instant Message

Send Features to a Stream Service

Write to a CSV or JSON File .csv

WS

im

HTTP

ActiveMQ

Esri

Gallery Cursor-on-Target

Hadoop

Kafka

MongoDB

RabbitMQ

Twitter

you can create

your own

connectors

GeoEvent Server

Inp

uts

Ou

tpu

ts

GeoEvent Services

Message Formatter SMTP

Connectors TransportAdapter

Feature-JSON Kafka

Send an Email

Publish Feature-JSON to Kafka

Big Data Store

Transports

Feature Service

File

HTTP

HTTP+BasicAuth

HTTP+OAuth

SMTP

Stream Service

TCP

UDP

WebSocket

XMPP

Cache

Adapters

Feature-JSON

GeoJSON

JSON

Message Formatter

RSS

Text

WebSocket

ActiveMQ

Hadoop

IRC

Kafka

MongoDB

MQTT

Esri

Gallery

RabbitMQ

TCP-Squirt

Twitter

Cursor-on-Target

Esri

Gallery

SMS

Sending Real-Time DataOutput Connector = Adapter + Transport

Creating InputsInputs are configured using Connectors

GeoEvent Server

National Weather

Service

US Geological Survey

Live Traffic Services

Ou

tpu

ts

GeoEvent ServicesStorms

Earthquakes

Traffic

Connectors

Inputshttp://

RSS Connector

Cre

ate

In

pu

t

JSON over REST

Cre

ate

In

pu

t

ConnectorA Connector’s Purpose

• By choosing a Connector, the user implicitly selects components from

GeoEvent Server that know:

▪ HOW to move data (Transport)

▪ WHAT the data looks like (Adapter)

Example Input

TCP Transport Text Adapter

Input (Receive text from a TCP Socket)

Start

StopByte [ ]

GeoEventGeoEvent Services

GeoEvent Server

SDK

You can create your own custom transports, adapters and processors using the

GeoEvent Software Development Kit (SDK).

Extending GeoEventSoftware Development Kit (SDK)

Your processors

Your

connectors

Your

connectors

GeoEvent Server

Inp

uts

Ou

tpu

ts

GeoEvent Services

A note on semantics: the SDK is used to build transports, adapters and processors. Input and output

connectors are built by pairing a transport and adapter in GeoEvent Manager.

- api: JavaDoc content associated with GeoEvent Server SDK

- repository: Local maven repository

- samples: Sample processors, transports, adapters

- GeoEvent Server Developer Guide

Extending GeoEvent ServerSoftware Development Kit (SDK)

Transports

TransportWhat makes up a Transport?

Transport Definition • Defines transport metadata and properties

Transport • Implements transport application logic

Transport Service • Provides a Transport Definition

• Instantiates new Transports

• Transports are given

- Properties that define behavior

- A “ByteListener” where bytes should be sent

• Transport is started by the server and it sends bytes to the receiver

• Transport is stopped by the server and it stops sending bytes

Transport BehaviorTransports

Transport LifecycleTransports

Transports have a lifecycle that determines if they are producing data

STOPPED

STOPPING

STARTED

STARTING

ERROR

STOPPED

Server calls start() Transport reads

properties and starts

Server calls stop()Transport closes

resources

Internal error occurs

STARTED

• Outbound Transports accept arrays of bytes from the Adapter and transmit them.

• Occasionally the destination for the bytes depends on content in the GeoEvent.

- The Transport has the option of “looking back” at the GeoEvent that generated the bytes, and

using it to route the data.

- Email address

- Phone number

Outbound TransportsTransports

Adapters

AdapterWhat makes up an Adapter

Adapter • Implements adapter application logic

Adapter Definition • Defines adapter metadata and properties

Adapter Service • Provides an Adapter Definition

• Instantiates new Adapters

• Adapters are given

- Properties that define behavior

- A “GeoEventListener” where the GeoEvents should be sent

• Adapters are DATA DRIVEN

- No start/stop calls

- The adapter is handed a byte array and pushes any generated GeoEvents to the Listener

Adapter BehaviorAdapters

Custom

Connectors

- Use SDK to build transports or adapters

- Custom connector made by pairing transport with an adapter

- Both transport and adapter can be custom, or custom transport only or custom

adapter only

Custom Connectors With DevelopmentTransport + Adapter

- You actually may not need the SDK to make a custom connector

- Familiarize yourself with all out-of-box transports and adapters

- Out of box connectors do not cover all unique permutations of adapter and

transport pairings

- You may already have all building blocks to make your “custom” connector

Custom Connectors Without DevelopmentTransport + Adapter

Clone an existing connector & tailor it’s properties to make a more specific connector

Custom Connectors Without Development

Clone an existing connector & tailor it’s properties to make a more specific connector

Custom Connectors Without Development

Extend GeoEvent Server by downloading additional transports, adapters and processors

https://github.com/esri/solutions-geoevent-java

http://links.esri.com/geoevent-gallery

More Options on the GeoEvent Gallery

DEMO Custom Adapter:

regexText-adapter…developed using SDK

DEMOCustom Connector:

XML over TCP …no development required

Processors

ProcessorsWhat is a processor?

• Processors perform some action on each GeoEvent passed to it

• Processors can be used to:

- Modify existing fields or the geometry

- Add new fields

- Filter

- Create new GeoEvent(s)

- Perform GeoFencingGeoEvent Server

Inp

uts

Ou

tpu

ts

GeoEvent Services

Processors

Applying real-time analyticsGeoEvent Processing

You can create

your own

processors.

• You can perform continuous analytics on GeoEvents as they are received using

a processor.

GeoEvent Server

Inp

uts

Ou

tpu

ts

GeoEvent Services

Buffer Creator

Convex Hull Creator

Difference Creator

Envelope Creator

Field Calculator

Field Enricher

Field Mapper

Geotagger

Incident Detector

Intersector

Projector

Simplifier

Symmetric Difference

Track Gap Detector

Field Reducer Union Creator

Ou

t o

f th

e B

ox

Add XYZ

Esri

Gallery

Bearing

Ellipse

Event Volume Control

Extent Enricher

Field Grouper

GeoNames Lookup

Range Fan

Reverse Geocoder

Service Area Creator

Symbol Lookup

Track Idle Detector

Unit Converter

Visibility

Motion Calculator Query Report

ProcessorLifecycle Review

GeoEvent Server

GeoEvent Service

Processor

Validation

Shutdown Initialization

Creation

Processing…

Anatomy of a ProcessorWhat makes up a Processor?

• Implements processor lifecycle

Configuration • Defines processor, it’s dependencies and metadata

Processor Definition • Defines processor metadata, properties

and GeoEvent Definition(s)

Processor

Processor Service • Instantiates processor definition, processor and

applies configuration

Maven

- Maven is an Apache project for automated software building, dependency

management, and testing

- Using Maven in secure environments raises some unique and potentially

extremely frustrating challenges

- Internet connectivity is at the core of most Maven installations

- All of the GeoEvent SDK samples are Maven projects

- Recommended you start with an existing sample

- Using the GeoEvent SDK means using Maven

- It is possible set up a GeoEvent project without Maven, but it can be very very

frustrating!

For those of us in closed, secure environments…Maven

- So with all this caution, why and how would I even do this?

- It’s not that bad, just know these things:

• Read the Developer Guide, especially:

- “Overview of the Sample Projects”

- “Using Eclipse to Edit and Build Custom Components”

- “Starting a New Project”

- “Building and Deploying the Sample Projects”

- At least one sample in “The Sample Projects”

- Continued…

For those of us in closed, secure environments…Maven

• Build outside your work environment, where you have internet connectivity

- Use one of the samples in the SDK to get started – one from the Developer Guide

- This downloads dependencies to your local machine

- Burn your entire .m2 folder to CD

- Hidden by default

- C:\Users\username\.m2 on Widows

- /root/.m2 on Linux

- Place .m2 in equivalent location in secure environment

- Create .m2/settings.xml file and point to local repository

For those of us in closed, secure environments…Maven

- Find out if your agency or department has a repository manager

- Artifactory

- Sonatype

- Archiva

- If so, deploy the artifacts to the repository manager

- Update your local /.m2/settings.xml to reference the repository manager

For those of us in closed, secure environments…In-house Maven

Scripting “Upstream”

Scripting “upstream”Alternate pattern for ingesting real-time data

• What if I don’t know Java?

• What if I don’t have time to learn a new SDK?

• What if I don’t want to (maybe) have to re-compile my connector at each new

GeoEvent version?

Scripting “upstream”Alternate pattern for ingesting real-time data

• Possible and easy to write your own app

• Runs “upstream” from GeoEvent

• Consumes real-time feed

• Parses and transforms data

• Sends data to GeoEvent in easy out-of-box form, like csv

Scripting “upstream”CTfastrak

• GTFS = General Transit Feed Specification

- Common format for public transportation schedules

• GTFS-realtime

- Trip updates

- Alerts

- Vehicle positions

• CTfastrak

- Bus Rapid Transit system in Connecticut

- http://www.cttransit.com/about/developers/gtfsdata/

• GTFS-rt is now native to GeoEvent

• (But this slide section and demo still convey the point)

DEMOPython for CT Transit

GTFS-rt

https://github.com/Esri/public-transit-tools/tree/master/send-

GTFS-rt-to-GeoEvent

Summary

• Connectors get data in and out of GeoEvent.

• Connectors pair a transport and an adapter.

• You can develop your own transports and adapters.

• You can configure your own connectors by pairing transports and adapters (custom

developed or out-of-box).

• Processors perform analytics on geoevents.

• You can build your own processors, as well.

• Maven is a key component to development with the GeoEvent Server SDK.

• You can also employ alternate tactics for custom GeoEvent development, such as

writing scripts that send data to GeoEvent.

Where to learn more?Resources

• To learn more, visit the ‘Get Started’ area of the GeoEvent Server page:

- http://links.esri.com/geoevent

- Introduction

- Big Data Store

- Resiliency

- Multiple-Machine Sites

- Notifications

- Stream Services

- RSS, HTTP, Files

- REST Admin API

• Ask questions on the GeoEvent Forum:

- http://links.esri.com/geoevent-forum

Please Take Our Survey on the Esri Events App!

Download the Esri Events app and find

your event

Select the session you attended

Scroll down to find the feedback

section

Complete answersand select “Submit”

Questions?

GeoEvent Server:

A Developer's GuideMark Bramer

Esri Professional Services

Vienna, VA