Deploying Live Data with High Availability Servers

Paul DoddDerrick Burke

Overview

• Definitions

• Server Deployment Options

• Data Deployment Options

• Data Update Options and Architecture

• Looking at a Live Feed

http://www.esri.com/services/disaster-response

http://www.esri.com/services/disaster-response

Definitions – Clearing the Air

Live Data, what do we mean?

• Frequently Changing Content!

• Real-Time Data- Changes moment-by-moment- Trickle or burst- No set intervals

• Near Real-Time Data- Periodic or Batch changes- Bulk updates typical- Timed intervals

What Constitutes HA (High-Availability)?

• Wikipedia- An implementation that ensures a prearranged level of operational

performance during a contractual period of time. (paraphrased)

• Availability in Nine’s (scheduled and unscheduled downtime allowance per year)- Two.Five 9’s (99.5%) = ~44 Hours- Three 9’s (99.9%) = ~8 3/4 Hours- Four 9’s (99.99%) = ~7/8 Hour or ~53 minutes

• Performance and Monitoring- Predictable and Stable performance regardless of growth.- Up does not mean available – Monitor your end-points

Design Factors for HA(as it applies to ArcGIS)

• Target Audience- Limited Users (consistent) – Static or Controlled Growth- Public (unlimited and varying demand) – Auto Scaling

• Application or Behavior- Read/Only – Viewers or 3rd party source- Read/Write – Editors, Geoprocessing (CPU intensive)

Design Factors for HA (cont)(as it applies to ArcGIS)

• Data Requirements- Static or rarely updated – Cache or Custom Extract- Random updates – RDBMS- Frequently replaced or updated – Stream, RDBMS, or Custom- Complexity – How heavy is your data?

Architecting HA

• Limit Points of Failure- CPU – Multi-Site / Multi-Node- Storage – Fault Tolerance with multiple I/O paths- Network – Load Balanced with Failover

• Cloud-Based or Internal or Hybrid- Greater flexibility, less infrastructure- Factor Physical vs. Virtual cost

Architecting HA (cont)

• Plan for and test Backup / Failover strategies

• Keep it simple!!!- An over-complicated design can open the door to failures!

Server Deployment Options

ArcGIS Server - HA Deployments

• Multi-Machine Sites- Single-Cluster- Balance workload

• Multi-Machine Clustered Sites- Divide services by function / purpose- Can be chatty- Increased administration

• Simple / Dynamic Expansion• Shared Config Store• Scalable w/restrictions (shared resources)

ArcGIS Server - HA Deployments (cont)

• Single-Machine HA Sites (active-active)- Very scalable (cloud)- Quick expansion (cloud)- Local Config Store / Data- High Volume- Public Access- External Network Load Balancer (NLB)

• Controlled Administration- Structured Sites (staging / production)

• Advanced solution

ArcGIS Server - HA Deployments (cont)

• ArcGIS Online – Don’t forget!- Esri managed- Scaled- Load balanced- Minimal administration- Three 9’s (99.9) for Hosted Feature Services

Data Deployment Options

ArcGIS Server - Data Deployment Options

Locally Managed- File Geodatabase

- Read-Only Services- Local to each Server

- Enterprise Geodatabase- Read-Write Services- Centralized RDBMS

- Supports Failover and Redundancy- Localized RDBMS on each Server

- Redundant, but can be difficult to maintain

Cloud Managed- Microsoft Azure?- Amazon RDS?- ArcGIS Online

- Hosted Feature Services- Map Services

Data Update Options and Architecture

Updating Live Data – Formats

Real-Time Data- Sensors

- Vehicles- Manufacturing / Industrial- Field equipment- SCADA

- Internet Resources- Streaming (Twitter)- Wide Range of Formats- CSV, XML, KML, RSS, GeoRSS, …

Near Real-Time Data- File based

- Shapefile- CSV, XML, KML, RSS, GeoRSS, …- Raster / Spatial Analysis- NetCDF- BIN

- Bulk / Batch- Heavy Geoprocessing / Enrichment- Data Transformation- Data Aggregation (multiple sources)

Updating Live Data – Options

• GeoEvent Extension (GEE)- Selected Content

• Aggregated Live Feed (ALF) methodology

Updating Live Data(what do I choose?)

• SCADA System w/Alerts- GeoEvent Ext.

• Raster Weather Data- ALF – ArcPy w/Spatial Analyst

• Vehicle Tracking- GeoEvent Ext. w/GeoFence

• Shapefile Processing- ALF – ArcPy

Source / Function GEE ALFSensors Networks X *Vehicle Tracking X *

Raster XWeather Data * X

Shapefile XFile GDB X

XML / KML X XCSV X X

JSON X XRSS / GeoRSS X X

BIN XNetCDF X

Internet Streams X *Geoprocessing X

GeoFence X ** = Depends on format and / or architecture

ArcGIS Server GeoEvent Extension

• Data Bridge

• Update / Output- Feature Service- Expose Stream Service- ArcGIS Online- Others

• HA w/Clusters

Let’s dive into GeoEvent Extension

• Input Connector – Read

• Decision – Filter

• Compute – Processor

• Output Connector – Write

ArcGIS Server GeoEvent Extension (cont)

• GeoFence Alerting- Polygon(s)- Spatial Filter

• Position Tracking- Inside / Outside- Enter / Exit

Aggregated Live Feed (ALF) Methodology

• Python Solution• GP w/ArcPy

• Update / Output• Read-Only File GDBs• Feature Services• Others…Analysis

• Flexible & Extensible

• HA w/Redundancy

ALF Methodology – NDFD Precipitation, Raster Processing

http://tmservices1/arcgis/rest/services/LiveFeeds/NDFD_Precipitation/MapServer

http://tmservices1/arcgis/rest/services/LiveFeeds/NDFD_Precipitation/MapServer

ALF Methodology – Cloud / Distributed Feed

Internet

Log Files

WorkFileGDB

Feed Routine(Aggregator)

DeploymentLogic

Data

FileGDBArchive

Zip File

Feed AggregatorALFprocessor.py

Data DeploymentALFdeployer.py

Internet

WorkFolder

Download andDeployment

Logic

LiveFileGDB

ArcGIS Server

PublishedMap Service

AmazonS3

Common Storage(External or Internal)

ALF Feed Construction – What’s required

• Pre-requisites- Python- ArcPy- Others (.py, 7zip, degrib, …)

• Optional- IDE – PyScripter, VS, …

• Components- ALFlib.py

- Function / Class Library

- ALFprocessor.py- Feed Processor +- ‘.cfg’ (data processing logic)

- ALFdeployer.py- Archive Distribution

Let’s dive into a ALF Live Feed

Flickr: Toolmantim

http://www.flickr.com/photos/toolmantim/6170448143/

Questions & Comments?

• AGOL Community Page (samples, scripts) http://esriurl.com/LiveFeed

• GeoEvent Extension Resources: http://server.arcgis.com/en/geoevent-extension/

• Live Data Feeds : • http://blogs.esri.com/esri/arcgis/2014/07/09/new-live-feeds-

added/• http://tmservices1.esri.com/arcgis/rest/services/LiveFeeds

http://esriurl.com/LiveFeedhttp://server.arcgis.com/en/geoevent-extension/http://blogs.esri.com/esri/arcgis/2014/07/09/new-live-feeds-added/http://tmservices1.esri.com/arcgis/rest/services/LiveFeeds

Rate This Sessionwww.esri.com/RateMyDevSummitSession

pdodd@esri.comdburke@esri.com

@derrickburke

Thank You!

Slide Number 1Slide Number 2Slide Number 3OverviewSlide Number 5Slide Number 6Slide Number 7Definitions – Clearing the AirLive Data, what do we mean?What Constitutes HA (High-Availability)?Design Factors for HA� (as it applies to ArcGIS)Design Factors for HA (cont)� (as it applies to ArcGIS)Architecting HAArchitecting HA (cont)Server Deployment OptionsArcGIS Server - HA DeploymentsArcGIS Server - HA Deployments (cont)ArcGIS Server - HA Deployments (cont)Data Deployment OptionsArcGIS Server - Data Deployment OptionsData Update Options and ArchitectureUpdating Live Data – FormatsUpdating Live Data – OptionsUpdating Live Data� (what do I choose?)ArcGIS Server GeoEvent ExtensionLet’s dive into GeoEvent ExtensionArcGIS Server GeoEvent Extension (cont)Aggregated Live Feed (ALF) MethodologyALF Methodology – NDFD Precipitation, Raster ProcessingALF Methodology – Cloud / Distributed FeedALF Feed Construction – What’s requiredLet’s dive into a ALF Live FeedQuestions & Comments?Rate This SessionSlide Number 36