3saqs technical workshop april 30 th, 2015 data warehouse current and future development...
TRANSCRIPT
![Page 1: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/1.jpg)
3SAQS Technical WorkshopApril 30th, 2015
Data WarehouseCurrent and Future Development
Intermountain West Data Warehouse
![Page 2: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/2.jpg)
Outline• Update - DW Overview & Development • Data Request Status• Visualization tools
– Image browser link– Model-to-obs tool– Emissions review tool– Dynamic Geospatial Model visualization tool– ENVIRON Source Apportionment tool ENVIRON site
• Functionality• Data Request Form
• Prototype data selection form• Category Management
• Inventory Management• Forum & email notifications
• Wrap-up and action items
3SAQS Technical Workshop, CIRA, Fort Collins, CO April 30, 2015
![Page 3: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/3.jpg)
Outline• Update - DW Overview & Development • Data Request Status• Visualization tools
– Image browser link– Model-to-obs tool– Emissions review tool– Dynamic Geospatial Model visualization tool– ENVIRON Source Apportionment tool ENVIRON site
• Functionality• Data Request Form
• Prototype data selection form• Category Management
• Inventory Management• Forum & email notifications
• Wrap-up and action items
3SAQS Technical Workshop, CIRA, Fort Collins, CO April 30, 2015
![Page 4: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/4.jpg)
3SAQS Technical Workshop, CIRA, Fort Collins, CO April 30, 2015
IWDW Overview
![Page 5: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/5.jpg)
3SAQS Technical Workshop, CIRA, Fort Collins, CO April 30, 2015
IWDW Overview
![Page 6: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/6.jpg)
DW Development Update• Project Management– Q1 2015
• Data Requests; Inventory Management; Data Delivery• Documentation
• Data & Information Management– Q2 2015 through 2016
• Visualization tool development (modeling I/O & analysis)• Modeling platform review• Expanded documentation and “easy-to-reference” information
3SAQS Technical Workshop, CIRA, Fort Collins, CO April 30, 2015
![Page 7: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/7.jpg)
3SAQS Technical Workshop, CIRA, Fort Collins, CO April 30, 2015
![Page 8: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/8.jpg)
3SAQS Technical Workshop, CIRA, Fort Collins, CO April 30, 2015
![Page 9: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/9.jpg)
Outline• Update - DW Overview & Development • Data Request Status• Visualization tools
– Image browser link– Model-to-obs tool– Emissions review tool– Dynamic Geospatial Model visualization tool– ENVIRON Source Apportionment tool ENVIRON site
• Functionality• Data Request Form
• Prototype data selection form• Category Management
• Inventory Management• Forum & email notifications
• Wrap-up and action items
3SAQS Technical Workshop, CIRA, Fort Collins, CO April 30, 2015
![Page 10: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/10.jpg)
Data Request Summary – pg. 1Request Requestor Data
Supporting Docs
RequestSubmitted
3-State Steward
Action
Direct Interest Studies for WSAQS Cooperators
Methane/VOC Control Sensitivity Study
Tammy Thompson CIRA
2008 Emissions Scope of Work Yes
Emissions Inventory Data Summary for 2008
Tammy Thompson CIRA
2008 Emissions None Yes Mike Barna Filled 4/24/2015
Source Apportionment Modeling for Nitrogen in Greater Yellowstone Area (GYA)
Tammy Thompson CIRA
2011 Emissions(unreleased)
Project Plan Yes Mike Barna
National Park AQRVTodd McDonnell E&S Environmental Chemistry
2011 Emissions(unreleased)
None Yes
Routes to Sustainability for Natural Gas Development and Water and Air Resources in the Rocky Mountain Region - Sensitivity Analysis
Shannon Capps University of Colorado, Boulder
2008bBase year Platform (partial)
Modeling Protocol and Timeline
YesKevin Briggs
Filled 4/24/2015
Comparison of source contributions to nitrogen deposition at Class 1&2 areas in the Western U.S.
Tammy Thompson CIRA
2008b Emissions and CAMx inputs
None No
WRAP TSS Emission Tool Updates – 2011a emissions displays
Cassie Archuleta Air Resource Specialists
2011a Emissions (unreleased)
None No Tom Moore
Inventory--categories and methodology consistent with previous WRAP inventories for regional haze from 2004 and 2008 (reasonable progress)
Daniel Bon CDPHE2008b Base year Platform
None (may be taken care under Archuleta request above)
Yes Tom Moore
Will fulfill via TSS after update w/2011 emissions
![Page 11: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/11.jpg)
Data Request Summary – pg. 2
Request Requestor DataSupporting
DocsRequest
Submitted3-State
StewardAction
NEPA Studies
Hiawatha Regional Energy Development Project (HREDP) Environmental Impact Statement
Ralph Morris ENVIRON
2008b NEPA Platform
None No Charis Tuers
Converse County Development Environmental Impact Statement (EIS)
Chao-Jung Chien AECOM
2008b NEPAPlatform
None YesRyan McCammon
Fulfilled3/18/2015
Outside Requests (applicability to WSAQS Cooperators not clear)
Colorado Oil and Gas Sensitivity Analyses
James Wilkinson Golder Associates Inc.
2008b NEPA Platform
None Yes
Update of a USFS Fire handbook Suresh Ratnam University of Southern California
Base2020_08a None No
Improving Ozone modeling in Uintah Basin
Huy Tran Utah State University
2008b Emissions and utilities
NoneYes incomplete
Analysis of FRAPPÉ and DISCOVER-AQ 2014 campaigns
Gabriele Pfister NCAR
2011a Emissions and utilities(unreleased)
None Yes
![Page 12: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/12.jpg)
Responding to Data Requests
• Process needs for operating the Data Warehouse noted so far:– the identities, roles, and responsibilities of
cooperating agency “stewards” need to be clarified– the necessary level of documentation and
applicability to the overall goals of the cooperators; and
– the form of a “release” of a platform and its component data, including the associated use recommendation.
![Page 13: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/13.jpg)
Pending Project Proposals with no data requests
Information or Data Request Proposal
Requestor Start Date DataRequest
SubmittedFulfillment Approach Action
Emissions QA 2011 Base A Gail Tonnesen EPA Region 8
10/1/2014 None No Internal project report completed
Ozone spatial plots for 2011 MPE Gail Tonnesen EPA Region 8
10/16/2014 None NoDisplay capability on DW in
development
PEM integration with SMOKE/CMAQ
Private consultant 11/11/2014 None No Cancelled
2011a Modeling Sensitivities Zac Adelman UNC-IE 9/1/2014 None NoConducted as internal DW modeling study analysis
PMDETAIL Ralph Morris ENVIRON
12/1/2014 None NoPending, will utilize files from DW and PMDETAIL project
2020 Future Case Sensitivity Mike Barna National Park Service, Air Resource Division
1/23/2015 None No Pending
![Page 14: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/14.jpg)
Outline• Update - DW Overview & Development • Data Request Status• Visualization tools
– Image browser link– Model-to-obs tool– Emissions review tool– Dynamic Geospatial Model visualization tool– ENVIRON Source Apportionment tool ENVIRON site
• Functionality• Data Request Form
• Prototype data selection form• Category Management
• Inventory Management• Forum & email notifications
• Wrap-up and action items
3SAQS Technical Workshop, CIRA, Fort Collins, CO April 30, 2015
![Page 15: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/15.jpg)
Image Browser• http://views.cira.colostate.edu/tsdw/Tools/ImageBrowser.aspx?pathid=TsdwPlotsRoot
• Displays large image sets delivered by data provider
• Possible Next Steps– Improve navigation (fewer mouse clicks)– Display meta data– Browse multiple image sets side by side
![Page 16: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/16.jpg)
Model-To-Obs Tool• http://views.cira.colostate.edu/TSDW/modeling/modelobscomparison_iwdw.aspx
• Compare model values to observed values by network
• Possible Next Steps– Platform selection– Additional filtering capabilities
![Page 17: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/17.jpg)
Emissions Review Tool
• http://views.cira.colostate.edu/tsdw/Emissions/ReviewTool.aspx
• View modeled emissions by source category, state, county, and SCC
• Possible Next steps– Refactor data base to correctly label and
aggregate parameters and source categories– Expand SCC visualization capabilities
![Page 18: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/18.jpg)
Dynamic Geospatial Model Visualization• http://views.cira.colostate.edu/tsdw_Dev/Maps/ModelDataGIS.aspx• Displays model output data with overlaid monitor data
![Page 19: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/19.jpg)
Dynamic Geospatial Model Visualization• ArcSDE (Spatial Database Engine) with SQL Server– Stores/Serves rasters of model data– Stores/Serves raster attributes– Data can be accessed remotely by ArcGIS users
• ArcServer– Provides web services which render data stored in
SDE– Web Services consumed by web applications such as
the data warehouse
![Page 20: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/20.jpg)
Dynamic Geospatial Model Visualization
• Advantages– Map Projection– Uniform color scale– Monitor data overlay– Ease of combining multiple features– Sharing data with other GIS users
![Page 21: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/21.jpg)
Dynamic Geospatial Model Visualization
• Disadvantages– More Pre-processing– Data duplication– Infrastructure heavy (ArcServer, ArcSDE)
![Page 22: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/22.jpg)
Dynamic Geospatial Model Visualization
• Possible Next steps– Parameterize, platform, species, layer,
time window– Additional Monitor data– Additional features• Cities• Roads
– Automate preprocessing, execute on demand
![Page 23: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/23.jpg)
Source Apportionment Tool• http://ims.environcorp.com/aqms_test• Developments/Status
• Received prototype from Environ• Database up and running
• Possible Next steps• Deploy prototype user interface on CIRA
systems• Integrate user interface with main warehouse
system• Integrate database with main warehouse DB• Automate data ingestion
![Page 24: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/24.jpg)
Outline• Update - DW Overview & Development • Data Request Status• Visualization tools
– Image browser link– Model-to-obs tool– Emissions review tool– Dynamic Geospatial Model visualization tool– ENVIRON Source Apportionment tool ENVIRON site
• Functionality• Data Request Form
• Prototype data selection form• Category Management
• Inventory Management• Forum & email notifications
• Wrap-up and action items
3SAQS Technical Workshop, CIRA, Fort Collins, CO April 30, 2015
![Page 25: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/25.jpg)
Data Request Form• Automated FTP fulfillment– One click FTP user creation and data access– Create/Update FTP user account– Enable FTP access to requested data without
duplication of data– Two Successful deliveries via FTP– Image Browser uses FTP fulfillment to serve
images
![Page 26: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/26.jpg)
Data Request Form• http://views.cira.colostate.edu/tsdw_dev/DataRequest/PlatformSelection.aspx
• Component Selection– Flattened tree– More descriptive information– Alternative to category tree
![Page 27: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/27.jpg)
Data Request Form• Possible Next Steps– Integrate component selection with request form– Possibly separate project proposal from data
request– Continue to evolve approval and stewardship
policies and procedures– Integrate data release status and/or evaluation
level– Notification email clean-up
![Page 28: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/28.jpg)
Category Management• Create/Remove/Edit
– Data Categories– File Items in category– Meta Data
• Browse file system• Data Management Reports
– Missing data• Possible Next steps
– Additional reporting as needed– Better display of category/file meta data parameters– File/Category history
![Page 29: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/29.jpg)
Inventory Management• MD5 checksums stored in relational database• History of file meta data
– file size– mod time– checksum
• Data Recovery tool• Next Steps
– Incorporate detailed meta data scraper– Automation of Checksum computation– Reporting
• Report changes to data requestors• File level data status
![Page 30: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/30.jpg)
Forum & Email Notifications• http://views.cira.colostate.edu/tsdw/Forum/
![Page 31: 3SAQS Technical Workshop April 30 th, 2015 Data Warehouse Current and Future Development Intermountain West Data Warehouse](https://reader033.vdocuments.mx/reader033/viewer/2022051620/56649e865503460f94b88a99/html5/thumbnails/31.jpg)
Outline• Update - DW Overview & Development • Data Request Status• Visualization tools
– Image browser link– Model-to-obs tool– Emissions review tool– Dynamic Geospatial Model visualization tool– ENVIRON Source Apportionment tool ENVIRON site
• Functionality• Data Request Form
• Prototype data selection form• Category Management
• Inventory Management• Forum & email notifications
• Wrap-up and action items
3SAQS Technical Workshop, CIRA, Fort Collins, CO April 30, 2015