Automating

Everything with FME

Agenda

• What is Automation?

• Data Loading

Scenarios– On Demand

– Scheduled

– Real Time

• Working with Big Data

• Making Data Available

The Journey of Data Superheroes

Episode #1:

A New Movement

I have systems that aren't connected and data lying

around. I need to start moving data.

The Journey of Data Superheroes

Episode #1:

A New Movement

I have systems that aren't connected and data lying

around. I need to start moving data.

Episode #2:

Transformation Menace

These systems are too different and I can’t easily move

data between them.

The Journey of Data Superheroes

Episode #1:

A New Movement

I have systems that aren't connected and data lying

around. I need to start moving data.

Episode #2:

Transformation Menace

These systems are too different and I can’t easily move

data between them.

Episode #3:

Validation Strikes Back

The data in these systems sucks. I need to filter out the

bad, fix it, and make sure only good stuff gets moved.

The Journey of Data Superheroes

Episode #1:

A New Movement

I have systems that aren't connected and data lying

around. I need to start moving data.

Episode #2:

Transformation Menace

These systems are too different and I can’t easily move

data between them.

Episode #3:

Validation Strikes Back

The data in these systems sucks. I need to filter out the

bad, fix it, and make sure only good stuff gets moved.

Episode #4:

Automation Awakens

I want data to be moved automatically. It should

happen each night, as soon as data is available, or on-

demand if someone needs it right then.

FME Server is automation.

What is Automation?

Running an

FME WorkspaceDelivering DataTriggering an event

Your Top 3 Automations*

1. Loading Data

2. Processing Large Data

Volumes

3. Making Data Available

* based on 5 years of survey data from FME Server Customers

Loading Data• On Demand

• Scheduled

• In Real Time

Example: Loading On Demand

Example: Scheduled Replication

“FME Server is the caretaker that lets us pay attention to other things.”

Example: Loading in Real Time

Data Loading Tips

✓ Validate data before it’s loaded and

immediately report the quality of a user’s

submission.

✓ For duplicate systems, nightly replication

avoids disruptions during the day.

✓ Keeping systems connected in real time is

better for insight/transparency.

Processing Data Volumes.

Large

Colossal

Big

Gigantic

Huge

Enormous

Gargantuan

Big Data Symptoms

❏ Difficulty figuring out how to handle all your data.

❏ So much data but not enough ability to process it.

❏ Regular use of the term terabyte.

❏ Work with imagery, LiDAR, networks,

live streams.

❏ Performing tasks like modelling, spatial

relationships, change detection, large-

scale map production.

Example: Big Networks (10,000 route queries!)

Example: Big Maps

“The volume of data required to be processed needed an automation workspace run on FME Server.”

Big Data Tips

✓ Look for ways to improve performance in

your FME workspaces.

✓ Use a dedicated FME Server machine.

✓ Use 64-bit FME for full access to memory

resources.

✓ Use multiple FME Engines for concurrent

processing.

✓ Use multiple machines, if you got ‘em.

✓ Estimate the total time and frequency to

decide if automation is worth it.

Example: Big Maps in the Cloud

Tip: Put Big Data in the Cloud.

✓ Scalable ● On-demand processing power.

✓ Reliable● 99.99%+ uptime.

✓ High performance● On-demand upgrade of processing power.

✓ Cost effective● Pay only for what you need.

22 Minutes to Destination

Example: Real-time Networks

Brake pressure OK

Fuel Level at 60%

JMS

“Fast Data” Tips

✓ Use multiple FME Engines for

scalability.

✓ Use message queue software

(JMS, SQS).

✓ Keep FME Engines always

running.

Making Data Available.

Consider: Do you need a

map focus, or a

metadata focus?

Consider: Can you offer

more than just the data?

● FME Server was created for automation.

● Automation can mean event triggers, running a

workflow on a schedule, or making data available on

demand.

● Leverage the full processing power of FME Desktop, FME

Server, and FME Cloud to get the most out of your

automation scenarios.

More info about all of the stories seen here:

● On-Demand Data Loading example (Swedish EPA): http://www.safe.com/fmeuc/automated-quality-controls-with-fme/● Scheduled Replication example (San Jose): http://www.safe.com/customers/san-jose-water-company/● Data Loading in Real Time example (San Jose): http://www.safe.com/presentation/oracle-database-triggers-and-fme-server-for-

task-tracking/● Big Networks example (London Fire Brigade): http://www.1spatial.com/wp-

content/uploads/2015/11/London_Fire_Case_Study_FINAL.pdf● Big Maps example: http://www.safe.com/presentation/national-map-production-with-mapnik-rasterizer/ ;

http://www.imgs.ie/ordance-survey-ireland● Big maps in the cloud example (Weather): http://www.slideshare.net/SafeSoftware/fme-around-the-world-46744668/53 ;

http://blog.safe.com/2015/02/event-driven-architecture-aws-lambda-s3/● “Fast data” railway example: http://blog.safe.com/2014/12/125000-mph-railways/● On-demand analytics: https://prezi.com/33sm_x4x6wvz/fme-on-demand-and-automated-spatial-reporting/● Open data, map focus: http://mslapps.mt.gov/geographic_information/data/databundler/ui_cdp● Open data, metadata focus:

http://findit.york.ca/opendata/portal/#/search?disp=f3566bc6&view=card&sortdir=desc&fq=fss_Categories:Spatial%5C%20Data

Thank you!

Questions?