Deploying Live Data with High Availability Servers

Paul DoddDerrick Burke

Overview

Definitions

Server Deployment Options

Data Deployment Options

Data Update Options and Architecture

Looking at a Live Feed

http://www.esri.com/services/disaster-response

http://www.esri.com/services/disaster-response

Definitions Clearing the Air

Live Data, what do we mean?

Frequently Changing Content!

Real-Time Data- Changes moment-by-moment- Trickle or burst- No set intervals

Near Real-Time Data- Periodic or Batch changes- Bulk updates typical- Timed intervals

What Constitutes HA (High-Availability)?

Wikipedia- An implementation that ensures a prearranged level of operational

performance during a contractual period of time. (paraphrased)

Availability in Nines (scheduled and unscheduled downtime allowance per year)- Two.Five 9s (99.5%) = ~44 Hours- Three 9s (99.9%) = ~8 3/4 Hours- Four 9s (99.99%) = ~7/8 Hour or ~53 minutes

Performance and Monitoring- Predictable and Stable performance regardless of growth.- Up does not mean available Monitor your end-points

Design Factors for HA(as it applies to ArcGIS)

Target Audience- Limited Users (consistent) Static or Controlled Growth- Public (unlimited and varying demand) Auto Scaling

Application or Behavior- Read/Only Viewers or 3rd party source- Read/Write Editors, Geoprocessing (CPU intensive)

Design Factors for HA (cont)(as it applies to ArcGIS)

Data Requirements- Static or rarely updated Cache or Custom Extract- Random updates RDBMS- Frequently replaced or updated Stream, RDBMS, or Custom- Complexity How heavy is your data?

Architecting HA

Limit Points of Failure- CPU Multi-Site / Multi-Node- Storage Fault Tolerance with multiple I/O paths- Network Load Balanced with Failover

Cloud-Based or Internal or Hybrid- Greater flexibility, less infrastructure- Factor Physical vs. Virtual cost

Architecting HA (cont)

Plan for and test Backup / Failover strategies

Keep it simple!!!- An over-complicated design can open the door to failures!

Server Deployment Options

ArcGIS Server - HA Deployments

Multi-Machine Sites- Single-Cluster- Balance workload

Multi-Machine Clustered Sites- Divide services by function / purpose- Can be chatty- Increased administration

Simple / Dynamic Expansion Shared Config Store Scalable w/restrictions (shared resources)

ArcGIS Server - HA Deployments (cont)

Single-Machine HA Sites (active-active)- Very scalable (cloud)- Quick expansion (cloud)- Local Config Store / Data- High Volume- Public Access- External Network Load Balancer (NLB)

Controlled Administration- Structured Sites (staging / production)

Advanced solution

ArcGIS Server - HA Deployments (cont)

ArcGIS Online Dont forget!- Esri managed- Scaled- Load balanced- Minimal administration- Three 9s (99.9) for Hosted Feature Services

Data Deployment Options

ArcGIS Server - Data Deployment Options

Locally Managed- File Geodatabase

- Read-Only Services- Local to each Server

- Enterprise Geodatabase- Read-Write Services- Centralized RDBMS

- Supports Failover and Redundancy- Localized RDBMS on each Server

- Redundant, but can be difficult to maintain

Cloud Managed- Microsoft Azure?- Amazon RDS?- ArcGIS Online

- Hosted Feature Services- Map Services

Data Update Options and Architecture

Updating Live Data Formats

Real-Time Data- Sensors

- Vehicles- Manufacturing / Industrial- Field equipment- SCADA

- Internet Resources- Streaming (Twitter)- Wide Range of Formats- CSV, XML, KML, RSS, GeoRSS,

Near Real-Time Data- File based

- Shapefile- CSV, XML, KML, RSS, GeoRSS, - Raster / Spatial Analysis- NetCDF- BIN

- Bulk / Batch- Heavy Geoprocessing / Enrichment- Data Transformation- Data Aggregation (multiple sources)

Updating Live Data Options

GeoEvent Extension (GEE)- Selected Content

Aggregated Live Feed (ALF) methodology

Updating Live Data(what do I choose?)

SCADA System w/Alerts- GeoEvent Ext.

Raster Weather Data- ALF ArcPy w/Spatial Analyst

Vehicle Tracking- GeoEvent Ext. w/GeoFence

Shapefile Processing- ALF ArcPy

Source / Function GEE ALFSensors Networks X *Vehicle Tracking X *

Raster XWeather Data * X

Shapefile XFile GDB X

XML / KML X XCSV X X

JSON X XRSS / GeoRSS X X

BIN XNetCDF X

Internet Streams X *Geoprocessing X

GeoFence X ** = Depends on format and / or architecture

ArcGIS Server GeoEvent Extension

Data Bridge

Update / Output- Feature Service- Expose Stream Service- ArcGIS Online- Others

HA w/Clusters

Lets dive into GeoEvent Extension

Input Connector Read

Decision Filter

Compute Processor

Output Connector Write

ArcGIS Server GeoEvent Extension (cont)

GeoFence Alerting- Polygon(s)- Spatial Filter

Position Tracking- Inside / Outside- Enter / Exit

Aggregated Live Feed (ALF) Methodology

Python Solution GP w/ArcPy

Update / Output Read-Only File GDBs Feature Services OthersAnalysis

Flexible & Extensible

HA w/Redundancy

ALF Methodology NDFD Precipitation, Raster Processing

http://tmservices1/arcgis/rest/services/LiveFeeds/NDFD_Precipitation/MapServer

http://tmservices1/arcgis/rest/services/LiveFeeds/NDFD_Precipitation/MapServer

ALF Methodology Cloud / Distributed Feed

Internet

Log Files

WorkFileGDB

Feed Routine(Aggregator)

DeploymentLogic

Data

FileGDBArchive

Zip File

Feed AggregatorALFprocessor.py

Data DeploymentALFdeployer.py

Internet

WorkFolder

Download andDeployment

Logic

LiveFileGDB

ArcGIS Server

PublishedMap Service

AmazonS3

Common Storage(External or Internal)

ALF Feed Construction Whats required

Pre-requisites- Python- ArcPy- Others (.py, 7zip, degrib, )

Optional- IDE PyScripter, VS,

Components- ALFlib.py

- Function / Class Library

- ALFprocessor.py- Feed Processor +- .cfg (data processing logic)

- ALFdeployer.py- Archive Distribution

Lets dive into a ALF Live Feed

Flickr: Toolmantim

http://www.flickr.com/photos/toolmantim/6170448143/

Questions & Comments?

AGOL Community Page (samples, scripts) http://esriurl.com/LiveFeed

GeoEvent Extension Resources: http://server.arcgis.com/en/geoevent-extension/

Live Data Feeds : http://blogs.esri.com/esri/arcgis/2014/07/09/new-live-feeds-

added/ http://tmservices1.esri.com/arcgis/rest/services/LiveFeeds

http://esriurl.com/LiveFeedhttp://server.arcgis.com/en/geoevent-extension/http://blogs.esri.com/esri/arcgis/2014/07/09/new-live-feeds-added/http://tmservices1.esri.com/arcgis/rest/services/LiveFeeds

Rate This Sessionwww.esri.com/RateMyDevSummitSession

pdodd@esri.comdburke@esri.com

@derrickburke

Thank You!

Slide Number 1Slide Number 2Slide Number 3OverviewSlide Number 5Slide Number 6Slide Number 7Definitions Clearing the AirLive Data, what do we mean?What Constitutes HA (High-Availability)?Design Factors for HA (as it applies to ArcGIS)Design Factors for HA (cont) (as it applies to ArcGIS)Architecting HAArchitecting HA (cont)Server Deployment OptionsArcGIS Server - HA DeploymentsArcGIS Server - HA Deployments (cont)ArcGIS Server - HA Deployments (cont)Data Deployment OptionsArcGIS Server - Data Deployment OptionsData Update Options and ArchitectureUpdating Live Data FormatsUpdating Live Data OptionsUpdating Live Data (what do I choose?)ArcGIS Server GeoEvent ExtensionLets dive into GeoEvent ExtensionArcGIS Server GeoEvent Extension (cont)Aggregated Live Feed (ALF) MethodologyALF Methodology NDFD Precipitation, Raster ProcessingALF Methodology Cloud / Distributed FeedALF Feed Construction Whats requiredLets dive into a ALF Live FeedQuestions & Comments?Rate This SessionSlide Number 36