Data Validation using FME Server

Clay Wise, OCP, GISPEnterprise ArchitectHampton Roads Sanitation District

HRSD (Hampton Roads Sanitation District

Who? What you should come away with

Whistler, BC to Virginia Beach, Virginia = 5,107 km or 3,173 miles

HRSD Mission Statement

!(

!(

!(

!(

!(

!(

!(

!(!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(!(

!(

!(

!(

!(

!(!(

!( !(

!(

!(

!(

!(

!(

!(

!(

!(!(

!(

!(

!(

!(!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!( !(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!( !(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!( !(

!( !(!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!( !(

!( !(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!( !(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!( !(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(!(

!(

!(

!(

!(

!(

!(

!( !(

!(

!(

!(

!(

!(!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!( !(!(

!( !(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(

!(!(

!(

!(

!(

!(

#

#

#

#

#

3SRP

HO

LLAND

RD

DAM NECK RD

LON

DO

N B

RID

GE

RD

SHIPPS CORNER RD

SW

ALLO

W D

R

WARWICK DR

TAYLOR FARM RD

HAWK AVE

SK

YLA

RK

DR

OL

D C

LUB

HO

US

E R

D

BAY

HE

AD

DR

BLU

E JA

Y D

R

LYN

NH

AVE

N P

KY

DR

AK

ES

MIL

E R

D

DORY DR

FINCH AVE

RAVEN AVE

CRUSADER CIR

STALLS WAY

PASTURE LN

OLD GLORY RD

NEW LAND DR

SO

NIC

DR

CA

NA

RY

DR

CA

PE

AN

N W

AY

DELAWARE XING

MONDRIAN LOOPVAN LOEN D

R

SCARBOROUGH WAY

WH

ITE

BIR

CH

LN

CR

EEKSIDE D

R

CAPE ANN CT

WIN

DY

WILLO

W C

T

PEELE CTSAXON PL

ENG

AGEM

ENT

CT

SPRUCE PINE RD

DUNNBURY CT

DUNNEBROOK DR

VERMEER DR

FOREST GREEN DR

RU

DD

Y O

AK

CT

WIMBLEDON W

AY

SADDLE CT

REDGROVE CT

MANOR GLEN CT

BARLBOROUGH WAY

YORKBOROUGH WAY

DAM NECK RD

HO

LL

AN

D R

D

CAPE ANN CT

Cardinal Estates

Chesapeake Landing

Oceana East Industrial Park

Oceana South Industrial Park

HRSD SHIPPS CORNER PRS

3TP

3TP

3TP

3TP

3TP

3TP3TP

3TP3TP

3TP

3TP

3TP

3TP

SUFFOLK

MATHEWS

CHESAPEAKE

ISLE OF WIGHT

VIRGINIA BEACH

GLOUCESTER

KING WILLIAM

KING AND QUEEN

MIDDLESEX

JAMES CITY

HAMPTON

YORK COUNTY

NORFOLK

NEWPORT NEWS

POQUOSON

PORTSMOUTH

WILLIAMSBURG

Mission Prevent pollution, protect public health, and support community

development by providing wastewater collection and treatment

Quick Facts

Treatment of Sanitary Sewer for over 1.7 million people.

More than 500 miles of pipes, 6-66 inches in diameter

102 Pump Stations 9 Major Treatment

plants 231 million gallons per

day is treated Serving 17

cities/counties

Regulatory Requirements

Context: Sanitary Sewer Overflows Task: Regional Hydraulic Model

Mostly Pressurized System 14 Jurisdictional Data Inputs 14 Jurisdictional GIS Database Designs Numerous Data Inputs

Over 1,800 jurisdictional pump stations Over 6,500 miles of jurisdictional sewer mains Over 120,000 sewer structures (manholes, vaults, etc…)

Challenge: Data Sharing Jurisdictional Hydraulic Model and general GIS Data Inputs to

HRSD HRSD Regional Hydraulic Model Data to Jurisdictions Data Validation & Standardization, Data Updates & Corrections

Data Validation Tool

What are we validating?

Every attribute (Field) Specific values

(material, locality code) How many were correct

out of the total Is the data spatially

correct? Is the geometry valid? Are their connection

points valid?

Validation Checks

Connection Points Where a jurisdiction connects to HRSD Force Main

system. Checking to see if the IDs match between their submittal and the connection point table we maintain.

Spatially correct Zero length lines Self intersecting polygons **With 14 jurisdictions we have run into all levels

of GIS users. This check should be in your validation process regardless of how simply or complex this process is.

How do you upload and download to FME Server?

FME Workbench Desktop product to develop and publish

data validation workspaces to FME Server Once published the workspace can be run

through a web browser End users do not need an FME license to

run the translation Can also download the posted workspace

to make changes

DOWNLOAD.mp4

How are we validating data?

FME Workbench Desktop product to construct/develop the

validation process Simple upload to publish process to FME

Server

FME Server

Upload &Download

The “nuts and bolts” of data validation

Example Workspace

Tester Transformers

Custom Transformers (Green)

Testers

Looking for:

NULL values

Specific Values (Locality Code)

Range (Elevations)

Showing the results

Using the drop down menu you shouldset this parameter to an attribute that contains unique ids. Even if you don’t want to show these ids on the report it is still required.

cfg01PumpStations:Calls out a specific variable in the report web page.

Showing the results (continued..)

cfg01PumpStations:Calls out a specific variable in the report web page.

<li class="error_message">cfgFailCount of cfgFeatCount features in your dataset have blank PSID attribute values.<ul><li>The ids of the failed features are: cfgFailedIds.</li></ul></li><br/>

Web page editing

I use Dreamweaver (CS3), but you can use cheaper software such as UltraEdit to edit the HTML.

cfg01PumpStations: This is the same variable we called out in the previous slide. After the information goes through the validation process it is placed in this variable. You can change them to whatever name suits you as long as they match the between the workbench file, and the webpage.

Web page editing

Color coding the message displayed. Here I simply have red for an error, and green if all the features passed.

cfg01Nodes: This is the same type of variable I have spoken about in the previous slides.

Results

cfgPipes01

cfgPipes02

cfgPipes04

cfgPipes03

Results are grouped according to the needs of the modelers

Green for passed, Red for failure

Connection points in blue are included with the summary checks

RUN.mp4

Data Upload

Once all the data is correct, a data download page will appear

The jurisdiction will then be able to upload their data through our firewall

Identity Management

Giving the jurisdictions access to our data

Oracle Identity Management

Granular protection inside the HRSD firewall

Single Sign on Reduce Risk Used for many

other processes than just validation

Virginia BeachSuffolkChesapeakeHamptonNorfolkPoquosonPortsmouthWilliamsburgGloucesterIsle of WightJames CityKing and QueenKing WilliamMathewsMiddlesexYork

Suffolk FME Server Example

Suffolk will log in using OIM

They will then find the GIS Parcels, and Water Meters that they maintain

They will then run FME Server

FME Server will join their GIS data to the water consumption data, and output a zip file for their use

Lessons learned

Good first run of the Validation Tool in winter of 2008 Modelers provided feedback for the second run

Geometry Check – Complete Some jurisdictions didn’t have very clean data (i.e. Self

intersecting polygons, zero length lines) Brief summary at the beginning of each feature class –

Complete Summary of linear feet of pipe- Complete Connection point validation – Complete

Positive: HRSD has control over all changes.

Benefits

HRSD has control over all changes

HRSD is able to quickly see where jurisdictions are failing to meet the requirements Instead of weeks of manually checks on each feature class, the

data can be processed in minutes. Sharing the reports with the jurisdictions

Efficient process for keeping data current, and valid. Simple form to fill out. Non-GIS personnel can run the application

over the web. Get to come to cool conferences like this one and talk about it!

Questions

Questions?

For more information: Clay Wise: cwise@hrsd.com HRSD: www.hrsd.com Data validation: fmepedia.com

FME Server > F > FME Server Examples>D > Data Validation with FME Server