enterprise data warehouse - esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf ·...

33
Enterprise Data Warehouse Enterprise Data Warehouse PRESENTATION FOR ESRI USER CONFERENCE 2017 July 2017 ESRI USER CONFERENCE 2017 1

Upload: nguyentruc

Post on 03-Jul-2018

218 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

Enterprise Data WarehousePRESENTATION FOR ESRI USER CONFERENCE 2017

July 2017 ESRI USER CONFERENCE 2017 1

Page 2: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

Agenda▪ EDW Team Introduction▪ EDW Background/Introduction▪ How to find out about EDW Content Status▪ What is the U.S. Forest Service▪ EDW Governance▪ What is EDW▪ Dimensional Model▪ EDW Milestones▪ Data Publishing/Security▪ EDW Usage▪ EDW in Action▪ Communicating with the EDW Team

July 2017 ESRI USER CONFERENCE 2017 2

Page 3: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

EDW Team▪ Lorri Peltz-Lewis – NEW FS EDW Program Manager ([email protected])

▪ Vacant position – FS Data Specialist

▪ Contract support team:o Kathleen Gundry, Project Manager

o Jocelyn Leatherwood, Communications Lead

o Bryan McEnaney, FME – Spatial ETL developer

o Blythe Norris, Informatica – Tabular ETL developer

o Buddy Evans, Cognos – Business Intelligence administrator, reports developer

o Deborah van Vlack – Informatica administrator/architect, ETL developer

o Syed Ahmed – FME – Spatial ETL developer

o Sabrina Turner – metadata specialist

o Others as needed

July 2017 ESRI USER CONFERENCE 2017 3

Page 4: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

EDW Background/Introduction - Purpose and Benefits

▪ Provide a trusted source of information from FS authoritative sources

▪ Aggregate data from multiple FS programs and make it accessible

▪ Provide tools for data analysis and management reporting

▪ Provide public access to FS data

▪ Promote data quality

▪ Promote data standardization

▪ Reduce redundant data storage

4ESRI USER CONFERENCE 2017July 2017

Page 5: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

National Forest Boundaries

July 2017 ESRI USER CONFERENCE 2017 5

The US Forest Service manages and protects 154 national forests and 20 grasslands in 44 states and Puerto Rico. The agency’s mission is to sustain the health, diversity, and productivity of the nation’s forests and grasslands to meet the needs of present and future generations.

National Forests in Alaska (not to scale)

Page 6: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

Chief’s Office

National Forest SystemState & Private

Forestry Research &

Development

R3 R4 R5 R…R1 R2

ANF

RD3 RD1 RD2 RD3RD1 RD2

BNF CNF DNF ENF FNF

3 Deputy Chiefs

9 NFSRegions

154 National Forests and Grasslands

1 – 20 States

190 Million Acres

Numerous RangerDistrict Offices

Bulk of Data:Field Inventories, Surveys, Observations

High-level View of the US Forest Service Organization

July 2017 ESRI USER CONFERENCE 2017 6

Page 7: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

Forest Service by the numbers

• 193 million acres • 154 national forests • 20 national grasslands• 1 Tall Grass Prairie• 7 research units• 80 experimental forests• 9 national monuments • 36.6 million acres of wilderness• 57,000 miles of streams• 9,100 miles of Scenic Byways• 4,400 miles of Wild and Scenic Rivers

• 277,000 heritage sites• 4,300 campgrounds• 122 ski areas• 158,000 miles of trails• 371,000 miles of roads• 40,000 buildings• 27,000 recreation sites• 17,000 vehicles• 13,000 bridges• 1,700 dams• 34,250 employees in 750 locations

July 2017 ESRI USER CONFERENCE 2017 7

Page 8: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

Challenge: FS data is organized in a number of silos – by programs and field offices who have autonomy and are suspicious of “headquarters” initiatives

July 2017 ESRI USER CONFERENCE 2017 8

Page 9: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Data Governance

Reinvigorated a dormant data governance body to provide content review

In the absence of agency data governance, conscripted FS staff with a passion for data to serve on a board to review and approve content and designed an approval process

July 2017 ESRI USER CONFERENCE 2017 9

Governance Structure

Page 10: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

What is the EDW?

ESRI USER CONFERENCE 2017 10July 2017

Page 11: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

EDW provides one-stop shopping for standardized, current, trusted data from disparate sources to be used for analysis or for sharing with external partners

ESRI USER CONFERENCE 2017 11July 2017

Page 12: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

One Data Source – Many Publication Formats

ESRI USER CONFERENCE 2017 12July 2017

Page 13: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

The Dimensional Model

July 2017ESRI USER CONFERENCE 2017 13

Dimension Target StarAmount Assigned

Planned Accomplishment StarAmount Planned

Actual Accomplishment StarAmount Accomplished

Performance Measure

Date Fiscal Year Fiscal Year Actual Date

Funding Source

Special Initiatives

Administrative Unit

Proclaimed Unit

Geopolitical Units

Nationally Designated Areas

Watershed

Ownership

Location

Page 14: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

The Accomplishment Star

July 2017ESRI USER CONFERENCE 2017 14

Actual Accomplishment

Amount Accomplished

by Date

by Performance Measure

by Special Initiative

by Funding Source

by Administrative Unit (Region/Forest/Ranger District)

by Proclaimed Forest

by Geopolitical (State, County,

Congressional District

by Watershed

by Nationally Designated Areas (wilderness, W&SR,…)by

Ownershipby Location

Page 15: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

2008

• EDW recommended in Integrated Business Environment report

2009

• Requirements Initial design

2010

• Governance Board chartered; database and initial tools in Production; data loading started

2011

• ETL and BI licenses purchased; started registering map services with data.gov; initial staff assigned

2012

• Incremental buildout of tools and content

2013

• Installed BI tools,; designed web page; developed strategic plan

EDW Milestones

July 2017ESRI USER CONFERENCE 2017 15

Page 16: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

EDW Milestones (continued)

July 2017ESRI USER CONFERENCE 2017 16

FY 2014

-Initiated collaborative development

-engaged stakeholders in content prioritization

-Published data for public access - EDW data drives FS Interactive Visitor and Travel Maps

FY 2015

-Data published on data.gov

- Continued to expand content to meet prioritized business needs

-Expanded capabilities for data discovery

FY 2016

-Establish reputation as preferred source of trusted data from authoritative sources

-Build out BI management reporting capability, Portal for BI reports

-Continue adding data to meet agency priorities

Page 17: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

Data published on FS Geodata Clearinghouse via downloadable datasets and map services

July 2017ESRI USER CONFERENCE 2017 17

http://data.fs.usda.gov/geodata/edw/index.php

Page 18: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

Map services – includes syncable services (collector)!

July 2017ESRI USER CONFERENCE 2017 18

https://data.fs.usda.gov/geodata/edw/mapServices.php

https://apps.fs.usda.gov/fsgisx02/rest/services/EDW

Page 19: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

EDW data published on data.gov

July 2017ESRI USER CONFERENCE 2017 19

https://catalog.data.gov/organization/fs-fed-us

Page 20: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

Data Security

▪ With major focus on securing sensitive data, EDW design included a separate database for data that is extremely sensitive or data containing Personally Identifiable Information (PII).

▪ Processes to restrict access by only authorizing access to a limited set of FS employees and to encrypt data in transit and in storage.

▪ Agency programs make the decision during the governance phase regarding the level of access:

▪ Published for public access

▪ Published for internal FS use only

▪ Limited to agency staff on a need to know basis (such as HR and Finance information)

July 2017ESRI USER CONFERENCE 2017 20

Page 21: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

EDW – Trusted and Engaged

July 2017 ESRI USER CONFERENCE 2017 21

System of Record – EDW directly supports FS Systems of Record

Recognized as the trusted aggregation of data from authoritative sources:Current (data is refreshed regularly)Trusted (drawn from authoritative sources – systems of record)Standardized (has been approved as an FS or generally accepted standard

Overall Engagement with USFS Business Areas ~80%Office of the Chief = 71% Business Operations = 56% Chief Financial Officer = 66% National Forest System Deputy Areas = 99% Research & Development Deputy Area = 83% State & Private Forestry Deputy Area = 100%

Page 22: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

EDW - Who is Using It?

▪ USFS GMO estimates 8,000 users, plus 4,000 occasional users, 28% of USFS uses geospatial technology

▪ Natural Resource Management (NRM) Tools –Geospatial Interface roughly 3,000 USFS users, 50-70% content delivered by EDW

▪ Scientists almost 60% - Archaeologists, Biologists, Botanists, Hydrologists, Foresters, Geologists, Soil Scientists, Timber, etc.

▪ NRM – 100% to be delivered by EDW in soon

July 2017ESRI USER CONFERENCE 2017 22

Engineer, 4%

FAM, 9%

GIS, 11%

Management, 9%

Realty Management, 3%

Scientist, 59%

Support Staff, 5%

USFS Staff Using EDW -- April-May 2017Total Connections = 68,765Unique Connections = 2,714

Page 23: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

EDW Usage – When is it being used (internal customers)

▪ 80% increase in usage from Q1 2016 to Q1 2017 - almost 260,000 connections

▪ Seasonal usage:▪ Jan-Mar = 138,619 (2016); 264,144 (2017)

▪ Apr-Jun = 166,582 (2016); 309,048 (est. 2017)

▪ Jul-Sep = 191,295 (2016); 338,104 (est. 2017)

▪ Oct-Dec = 147,659 (2016); 285,275 (est. 2017)

▪ April-May usage:▪ Month = 68,764 connection

▪ Weekly = 14,330 to 17,937 connections

▪ Daily = 2,172 to 4,802 connections

▪ Duration per connection undetermined

July 2017ESRI USER CONFERENCE 2017 23

Page 24: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

Where you can see EDW data in actionhttps://data.fs.usda.gov/geodata/edw/ - FS Geodata Clearinghouse – Enterprise Data - map services, downloadable datasets, Data Extract Tool

https://catalog.data.gov/organization/fs-fed-us - EDW Data in data.gov

https://usfs.maps.arcgis.com/home/index.html - EDW Data in ArcGIS Online (AGOL) - to find EDW Content in AGOL –search on: USFSEnterpriseContent

https://www.fs.fed.us/ivm/ Interactive Visitor Map

http://apps.fs.fed.us/TravelAccess/ Travel Access Map

https://egp.nwcg.gov/egp/default.aspx Some EDW Layers in the Fire Enterprise Geospatial Portal

http://apps.fs.fed.us/ArcN/rest/services/EDW FS Internal map services

https://www.fs.fed.us/ecosystemservices/FS_Efforts/forests2faucets.shtml Forests to Faucets map

EDW datasets are published as read only sync-enabled feature services for use in Collector while offline. These feature services can be used as reference layers for any Collector data collection project on mobile devices. The services can be found and added to web maps by going to the ArcGIS REST Services Directory. For more information: EDW SharePoint site or Enterprise Map Services SharePoint site.

July 2017 ESRI USER CONFERENCE 2017 24

Page 25: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

5/4/2017 Presenter - Lorri Peltz-Lewis R3 GIS COORDINATORS CONFERENCE CALL 25

EDW

EDW – Systems of Records - Operations

Page 26: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

5/4/2017 Presenter - Lorri Peltz-Lewis R3 GIS COORDINATORS CONFERENCE CALL 26

EDW

Transactional DBs (NRM, INFRA, Units, etc.)

Extract, Transform, Load (ETL) FME

Tabular DBs (NRM, Finance, Programs, etc.)

Extract, Transform, Load (ETL) Informatica

Cognos Business Intelligence (BI)

Non-Business Hours Data Processing Heartbeat

FME ServerEvening/Weekend

ETL Processing

Page 27: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

5/4/2017 Presenter - Lorri Peltz-Lewis R3 GIS COORDINATORS CONFERENCE CALL 27

EDW

InformaticaEvening/Weekend

ETL Processing

FME ServerEvening/Weekend

ETL Processing

Transactional DBs (NRM, INFRA, Units, etc.)

Extract, Transform, Load (ETL) FME

Tabular DBs (NRM, Finance, Programs, etc.)

Extract, Transform, Load (ETL) Informatica

Cognos Business Intelligence (BI)

Non-Business Hours Data Processing Heartbeat

Page 28: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

5/4/2017 Presenter - Lorri Peltz-Lewis R3 GIS COORDINATORS CONFERENCE CALL 28

EDW

InformaticaEvening/Weekend

ETL Processing

FME ServerEvening/Weekend

ETL Processing

Cognos BIEvening/Weekend

Processing

Transactional DBs (NRM, INFRA, Units, etc.)

Extract, Transform, Load (ETL) FME

Tabular DBs (NRM, Finance, Programs, etc.)

Extract, Transform, Load (ETL) Informatica

Cognos Business Intelligence (BI)

Non-Business Hours Data Processing Heartbeat

Page 29: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

5/4/2017 Presenter - Lorri Peltz-Lewis R3 GIS COORDINATORS CONFERENCE CALL 29

EDW – Near Real-Time DataAggregated Authoritative Data

Non-Transactional DatabaseReflection of Unit Entered Data

EDW

Geospatial services – AGOL, DaaS, etc.

Direct Connections – Citrix, Esri Tools, etc.

Data.gov – Open Data Delivery

Data Downloads

Business Intel (BI) Reports

Business Hours Data

Consumption Heartbeat

Page 30: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

EDW - Growing Enterprise Dependencies

July 2017 ESRI USER CONFERENCE 2017 30

0

10

20

30

40

50

60

No

v-1

4

Dec

-14

Jan

-15

Feb

-15

Mar

-15

Ap

r-1

5

May

-15

Jun

-15

Jul-

15

Au

g-1

5

Sep

-15

Oct

-15

No

v-1

5

Dec

-15

Jan

-16

Feb

-16

Mar

-16

Ap

r-1

6

May

-16

Jun

-16

Jul-

16

Au

g-1

6

Sep

-16

Oct

-16

No

v-1

6

Dec

-16

Jan

-17

Feb

-17

Mar

-17

Ap

r-1

7

External Map Service

Transactions (Millions)

Enterprise Data Warehouse

Fiscal Year 2015-2017

These transactions represent the customer interactions with our Enterprise Map Services, both internally by USFS business areas (e.g., FIA, NRM, FHTET, FAM) as well as externally by our partners and members of the public.

Geospatial workflows are the heartbeat of hundreds of agency workflows and analysis

Page 31: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

Forest Data Publication Status

31July 2017 ESRI USER CONFERENCE 2017

EDW Data Publication Status EDW on AGOL search for: USFSEnterpriseContent

Page 32: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

Communicating with the EDW Team

▪ EDW Governance SharePoint site (Internal) – Propose a new dataset in the EDWhttps://ems-team.usda.gov/sites/fs-cio-edwts/SitePages/EDW%20Governance.aspx

▪ EDW Users Forum (Internal) – General discussion about EDW data

https://ems-team.usda.gov/sites/fs-cio-edwts/Lists/EDW%20Users%20Forum/Management.aspx

▪ EDW Data Quality (Internal) – Report data quality issues with EDW data

https://ems-team.usda.gov/sites/fs-cio-edwts/Lists/Data%20Quality/Submitter.aspx

▪ EDW Status Forum (Internal) – Report an operational issue with the EDW

https://ems-team.usda.gov/sites/fs-cio-edwts/Lists/EDW%20Status%20Forum/Submitter.aspx

▪ Public & Staff can also send email to [email protected]

32July 2017 ESRI USER CONFERENCE 2017

Page 33: Enterprise Data Warehouse - Esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf · Hours Data Processing Heartbeat FME Server Evening/Weekend ETL Processing. ... EDW

Enterprise Data Warehouse

Questions?

July 2017 ESRI USER CONFERENCE 2017 33