enterprise data warehouse - esriproceedings.esri.com/library/userconf/proc17/papers/2116_74.pdf ·...
TRANSCRIPT
Enterprise Data Warehouse
Enterprise Data WarehousePRESENTATION FOR ESRI USER CONFERENCE 2017
July 2017 ESRI USER CONFERENCE 2017 1
Enterprise Data Warehouse
Agenda▪ EDW Team Introduction▪ EDW Background/Introduction▪ How to find out about EDW Content Status▪ What is the U.S. Forest Service▪ EDW Governance▪ What is EDW▪ Dimensional Model▪ EDW Milestones▪ Data Publishing/Security▪ EDW Usage▪ EDW in Action▪ Communicating with the EDW Team
July 2017 ESRI USER CONFERENCE 2017 2
Enterprise Data Warehouse
EDW Team▪ Lorri Peltz-Lewis – NEW FS EDW Program Manager ([email protected])
▪ Vacant position – FS Data Specialist
▪ Contract support team:o Kathleen Gundry, Project Manager
o Jocelyn Leatherwood, Communications Lead
o Bryan McEnaney, FME – Spatial ETL developer
o Blythe Norris, Informatica – Tabular ETL developer
o Buddy Evans, Cognos – Business Intelligence administrator, reports developer
o Deborah van Vlack – Informatica administrator/architect, ETL developer
o Syed Ahmed – FME – Spatial ETL developer
o Sabrina Turner – metadata specialist
o Others as needed
July 2017 ESRI USER CONFERENCE 2017 3
Enterprise Data Warehouse
EDW Background/Introduction - Purpose and Benefits
▪ Provide a trusted source of information from FS authoritative sources
▪ Aggregate data from multiple FS programs and make it accessible
▪ Provide tools for data analysis and management reporting
▪ Provide public access to FS data
▪ Promote data quality
▪ Promote data standardization
▪ Reduce redundant data storage
4ESRI USER CONFERENCE 2017July 2017
Enterprise Data Warehouse
National Forest Boundaries
July 2017 ESRI USER CONFERENCE 2017 5
The US Forest Service manages and protects 154 national forests and 20 grasslands in 44 states and Puerto Rico. The agency’s mission is to sustain the health, diversity, and productivity of the nation’s forests and grasslands to meet the needs of present and future generations.
National Forests in Alaska (not to scale)
Enterprise Data Warehouse
Chief’s Office
National Forest SystemState & Private
Forestry Research &
Development
R3 R4 R5 R…R1 R2
ANF
RD3 RD1 RD2 RD3RD1 RD2
BNF CNF DNF ENF FNF
3 Deputy Chiefs
9 NFSRegions
154 National Forests and Grasslands
1 – 20 States
190 Million Acres
Numerous RangerDistrict Offices
Bulk of Data:Field Inventories, Surveys, Observations
High-level View of the US Forest Service Organization
July 2017 ESRI USER CONFERENCE 2017 6
Enterprise Data Warehouse
Forest Service by the numbers
• 193 million acres • 154 national forests • 20 national grasslands• 1 Tall Grass Prairie• 7 research units• 80 experimental forests• 9 national monuments • 36.6 million acres of wilderness• 57,000 miles of streams• 9,100 miles of Scenic Byways• 4,400 miles of Wild and Scenic Rivers
• 277,000 heritage sites• 4,300 campgrounds• 122 ski areas• 158,000 miles of trails• 371,000 miles of roads• 40,000 buildings• 27,000 recreation sites• 17,000 vehicles• 13,000 bridges• 1,700 dams• 34,250 employees in 750 locations
July 2017 ESRI USER CONFERENCE 2017 7
Enterprise Data Warehouse
Challenge: FS data is organized in a number of silos – by programs and field offices who have autonomy and are suspicious of “headquarters” initiatives
July 2017 ESRI USER CONFERENCE 2017 8
Data Governance
Reinvigorated a dormant data governance body to provide content review
In the absence of agency data governance, conscripted FS staff with a passion for data to serve on a board to review and approve content and designed an approval process
July 2017 ESRI USER CONFERENCE 2017 9
Governance Structure
Enterprise Data Warehouse
What is the EDW?
ESRI USER CONFERENCE 2017 10July 2017
Enterprise Data Warehouse
EDW provides one-stop shopping for standardized, current, trusted data from disparate sources to be used for analysis or for sharing with external partners
ESRI USER CONFERENCE 2017 11July 2017
Enterprise Data Warehouse
One Data Source – Many Publication Formats
ESRI USER CONFERENCE 2017 12July 2017
Enterprise Data Warehouse
The Dimensional Model
July 2017ESRI USER CONFERENCE 2017 13
Dimension Target StarAmount Assigned
Planned Accomplishment StarAmount Planned
Actual Accomplishment StarAmount Accomplished
Performance Measure
Date Fiscal Year Fiscal Year Actual Date
Funding Source
Special Initiatives
Administrative Unit
Proclaimed Unit
Geopolitical Units
Nationally Designated Areas
Watershed
Ownership
Location
Enterprise Data Warehouse
The Accomplishment Star
July 2017ESRI USER CONFERENCE 2017 14
Actual Accomplishment
Amount Accomplished
by Date
by Performance Measure
by Special Initiative
by Funding Source
by Administrative Unit (Region/Forest/Ranger District)
by Proclaimed Forest
by Geopolitical (State, County,
Congressional District
by Watershed
by Nationally Designated Areas (wilderness, W&SR,…)by
Ownershipby Location
Enterprise Data Warehouse
2008
• EDW recommended in Integrated Business Environment report
2009
• Requirements Initial design
2010
• Governance Board chartered; database and initial tools in Production; data loading started
2011
• ETL and BI licenses purchased; started registering map services with data.gov; initial staff assigned
2012
• Incremental buildout of tools and content
2013
• Installed BI tools,; designed web page; developed strategic plan
EDW Milestones
July 2017ESRI USER CONFERENCE 2017 15
Enterprise Data Warehouse
EDW Milestones (continued)
July 2017ESRI USER CONFERENCE 2017 16
FY 2014
-Initiated collaborative development
-engaged stakeholders in content prioritization
-Published data for public access - EDW data drives FS Interactive Visitor and Travel Maps
FY 2015
-Data published on data.gov
- Continued to expand content to meet prioritized business needs
-Expanded capabilities for data discovery
FY 2016
-Establish reputation as preferred source of trusted data from authoritative sources
-Build out BI management reporting capability, Portal for BI reports
-Continue adding data to meet agency priorities
Enterprise Data Warehouse
Data published on FS Geodata Clearinghouse via downloadable datasets and map services
July 2017ESRI USER CONFERENCE 2017 17
http://data.fs.usda.gov/geodata/edw/index.php
Enterprise Data Warehouse
Map services – includes syncable services (collector)!
July 2017ESRI USER CONFERENCE 2017 18
https://data.fs.usda.gov/geodata/edw/mapServices.php
https://apps.fs.usda.gov/fsgisx02/rest/services/EDW
Enterprise Data Warehouse
EDW data published on data.gov
July 2017ESRI USER CONFERENCE 2017 19
https://catalog.data.gov/organization/fs-fed-us
Enterprise Data Warehouse
Data Security
▪ With major focus on securing sensitive data, EDW design included a separate database for data that is extremely sensitive or data containing Personally Identifiable Information (PII).
▪ Processes to restrict access by only authorizing access to a limited set of FS employees and to encrypt data in transit and in storage.
▪ Agency programs make the decision during the governance phase regarding the level of access:
▪ Published for public access
▪ Published for internal FS use only
▪ Limited to agency staff on a need to know basis (such as HR and Finance information)
July 2017ESRI USER CONFERENCE 2017 20
Enterprise Data Warehouse
EDW – Trusted and Engaged
July 2017 ESRI USER CONFERENCE 2017 21
System of Record – EDW directly supports FS Systems of Record
Recognized as the trusted aggregation of data from authoritative sources:Current (data is refreshed regularly)Trusted (drawn from authoritative sources – systems of record)Standardized (has been approved as an FS or generally accepted standard
Overall Engagement with USFS Business Areas ~80%Office of the Chief = 71% Business Operations = 56% Chief Financial Officer = 66% National Forest System Deputy Areas = 99% Research & Development Deputy Area = 83% State & Private Forestry Deputy Area = 100%
Enterprise Data Warehouse
EDW - Who is Using It?
▪ USFS GMO estimates 8,000 users, plus 4,000 occasional users, 28% of USFS uses geospatial technology
▪ Natural Resource Management (NRM) Tools –Geospatial Interface roughly 3,000 USFS users, 50-70% content delivered by EDW
▪ Scientists almost 60% - Archaeologists, Biologists, Botanists, Hydrologists, Foresters, Geologists, Soil Scientists, Timber, etc.
▪ NRM – 100% to be delivered by EDW in soon
July 2017ESRI USER CONFERENCE 2017 22
Engineer, 4%
FAM, 9%
GIS, 11%
Management, 9%
Realty Management, 3%
Scientist, 59%
Support Staff, 5%
USFS Staff Using EDW -- April-May 2017Total Connections = 68,765Unique Connections = 2,714
Enterprise Data Warehouse
EDW Usage – When is it being used (internal customers)
▪ 80% increase in usage from Q1 2016 to Q1 2017 - almost 260,000 connections
▪ Seasonal usage:▪ Jan-Mar = 138,619 (2016); 264,144 (2017)
▪ Apr-Jun = 166,582 (2016); 309,048 (est. 2017)
▪ Jul-Sep = 191,295 (2016); 338,104 (est. 2017)
▪ Oct-Dec = 147,659 (2016); 285,275 (est. 2017)
▪ April-May usage:▪ Month = 68,764 connection
▪ Weekly = 14,330 to 17,937 connections
▪ Daily = 2,172 to 4,802 connections
▪ Duration per connection undetermined
July 2017ESRI USER CONFERENCE 2017 23
Enterprise Data Warehouse
Where you can see EDW data in actionhttps://data.fs.usda.gov/geodata/edw/ - FS Geodata Clearinghouse – Enterprise Data - map services, downloadable datasets, Data Extract Tool
https://catalog.data.gov/organization/fs-fed-us - EDW Data in data.gov
https://usfs.maps.arcgis.com/home/index.html - EDW Data in ArcGIS Online (AGOL) - to find EDW Content in AGOL –search on: USFSEnterpriseContent
https://www.fs.fed.us/ivm/ Interactive Visitor Map
http://apps.fs.fed.us/TravelAccess/ Travel Access Map
https://egp.nwcg.gov/egp/default.aspx Some EDW Layers in the Fire Enterprise Geospatial Portal
http://apps.fs.fed.us/ArcN/rest/services/EDW FS Internal map services
https://www.fs.fed.us/ecosystemservices/FS_Efforts/forests2faucets.shtml Forests to Faucets map
EDW datasets are published as read only sync-enabled feature services for use in Collector while offline. These feature services can be used as reference layers for any Collector data collection project on mobile devices. The services can be found and added to web maps by going to the ArcGIS REST Services Directory. For more information: EDW SharePoint site or Enterprise Map Services SharePoint site.
July 2017 ESRI USER CONFERENCE 2017 24
Enterprise Data Warehouse
5/4/2017 Presenter - Lorri Peltz-Lewis R3 GIS COORDINATORS CONFERENCE CALL 25
EDW
EDW – Systems of Records - Operations
Enterprise Data Warehouse
5/4/2017 Presenter - Lorri Peltz-Lewis R3 GIS COORDINATORS CONFERENCE CALL 26
EDW
Transactional DBs (NRM, INFRA, Units, etc.)
Extract, Transform, Load (ETL) FME
Tabular DBs (NRM, Finance, Programs, etc.)
Extract, Transform, Load (ETL) Informatica
Cognos Business Intelligence (BI)
Non-Business Hours Data Processing Heartbeat
FME ServerEvening/Weekend
ETL Processing
Enterprise Data Warehouse
5/4/2017 Presenter - Lorri Peltz-Lewis R3 GIS COORDINATORS CONFERENCE CALL 27
EDW
InformaticaEvening/Weekend
ETL Processing
FME ServerEvening/Weekend
ETL Processing
Transactional DBs (NRM, INFRA, Units, etc.)
Extract, Transform, Load (ETL) FME
Tabular DBs (NRM, Finance, Programs, etc.)
Extract, Transform, Load (ETL) Informatica
Cognos Business Intelligence (BI)
Non-Business Hours Data Processing Heartbeat
Enterprise Data Warehouse
5/4/2017 Presenter - Lorri Peltz-Lewis R3 GIS COORDINATORS CONFERENCE CALL 28
EDW
InformaticaEvening/Weekend
ETL Processing
FME ServerEvening/Weekend
ETL Processing
Cognos BIEvening/Weekend
Processing
Transactional DBs (NRM, INFRA, Units, etc.)
Extract, Transform, Load (ETL) FME
Tabular DBs (NRM, Finance, Programs, etc.)
Extract, Transform, Load (ETL) Informatica
Cognos Business Intelligence (BI)
Non-Business Hours Data Processing Heartbeat
Enterprise Data Warehouse
5/4/2017 Presenter - Lorri Peltz-Lewis R3 GIS COORDINATORS CONFERENCE CALL 29
EDW – Near Real-Time DataAggregated Authoritative Data
Non-Transactional DatabaseReflection of Unit Entered Data
EDW
Geospatial services – AGOL, DaaS, etc.
Direct Connections – Citrix, Esri Tools, etc.
Data.gov – Open Data Delivery
Data Downloads
Business Intel (BI) Reports
Business Hours Data
Consumption Heartbeat
Enterprise Data Warehouse
EDW - Growing Enterprise Dependencies
July 2017 ESRI USER CONFERENCE 2017 30
0
10
20
30
40
50
60
No
v-1
4
Dec
-14
Jan
-15
Feb
-15
Mar
-15
Ap
r-1
5
May
-15
Jun
-15
Jul-
15
Au
g-1
5
Sep
-15
Oct
-15
No
v-1
5
Dec
-15
Jan
-16
Feb
-16
Mar
-16
Ap
r-1
6
May
-16
Jun
-16
Jul-
16
Au
g-1
6
Sep
-16
Oct
-16
No
v-1
6
Dec
-16
Jan
-17
Feb
-17
Mar
-17
Ap
r-1
7
External Map Service
Transactions (Millions)
Enterprise Data Warehouse
Fiscal Year 2015-2017
These transactions represent the customer interactions with our Enterprise Map Services, both internally by USFS business areas (e.g., FIA, NRM, FHTET, FAM) as well as externally by our partners and members of the public.
Geospatial workflows are the heartbeat of hundreds of agency workflows and analysis
Enterprise Data Warehouse
Forest Data Publication Status
31July 2017 ESRI USER CONFERENCE 2017
EDW Data Publication Status EDW on AGOL search for: USFSEnterpriseContent
Enterprise Data Warehouse
Communicating with the EDW Team
▪ EDW Governance SharePoint site (Internal) – Propose a new dataset in the EDWhttps://ems-team.usda.gov/sites/fs-cio-edwts/SitePages/EDW%20Governance.aspx
▪ EDW Users Forum (Internal) – General discussion about EDW data
https://ems-team.usda.gov/sites/fs-cio-edwts/Lists/EDW%20Users%20Forum/Management.aspx
▪ EDW Data Quality (Internal) – Report data quality issues with EDW data
https://ems-team.usda.gov/sites/fs-cio-edwts/Lists/Data%20Quality/Submitter.aspx
▪ EDW Status Forum (Internal) – Report an operational issue with the EDW
https://ems-team.usda.gov/sites/fs-cio-edwts/Lists/EDW%20Status%20Forum/Submitter.aspx
▪ Public & Staff can also send email to [email protected]
32July 2017 ESRI USER CONFERENCE 2017
Enterprise Data Warehouse
Questions?
July 2017 ESRI USER CONFERENCE 2017 33