the noaa big data project overview
TRANSCRIPT
![Page 1: The NOAA Big Data Project Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062302/58ec83df1a28ab666f8b45ad/html5/thumbnails/1.jpg)
The NOAA
Big Data Project Overview
Amy Gaskins, Project Director
February 2016
![Page 2: The NOAA Big Data Project Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062302/58ec83df1a28ab666f8b45ad/html5/thumbnails/2.jpg)
What is the Big Data Project?
• An innovative approach to publishing NOAA’s vast data resources and positioning them near cost-efficient high performance computing, analytic, and storage services provided by the private sector
• High-quality environmental data at low cost
• Increased public access to taxpayer-funded data
• Opportunities for new data products and services
![Page 3: The NOAA Big Data Project Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062302/58ec83df1a28ab666f8b45ad/html5/thumbnails/3.jpg)
Why Start Now?
• NOAA’s projected data growth is exponential
• By 2020, we’ll have 160 petabytes of archived data
• The rate of data capture will only continue to increase
![Page 4: The NOAA Big Data Project Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062302/58ec83df1a28ab666f8b45ad/html5/thumbnails/4.jpg)
Why a CRADA?
• Existing NOAA infrastructure and funding cannot support growing amount of data or level of external demand
• CRADA is a low-risk mechanism for creating and testing a smaller version of the full market ecosystem
• Provides an iterative approach to dissemination without disrupting NOAA operations
• Allows for lessons learned and real-time modifications to the dissemination process
![Page 5: The NOAA Big Data Project Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062302/58ec83df1a28ab666f8b45ad/html5/thumbnails/5.jpg)
Why Cloud Services?
![Page 6: The NOAA Big Data Project Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062302/58ec83df1a28ab666f8b45ad/html5/thumbnails/6.jpg)
The BDP Dissemination Model
Traditional Data Request Big Data Project
![Page 7: The NOAA Big Data Project Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062302/58ec83df1a28ab666f8b45ad/html5/thumbnails/7.jpg)
Important CRADA Rules of the Road
• Valid for three years with annual renewal option
• Collaborators and/or NOAA may choose to terminate with 30 days’ notice
• Collaborators have non-exclusive access to NOAA data • All NOAA data is up for discussion (excluding ITAR-restricted
and national security sensitive data)
• Collaborators must provide users with equal access to NOAA data on equal terms
![Page 8: The NOAA Big Data Project Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062302/58ec83df1a28ab666f8b45ad/html5/thumbnails/8.jpg)
The partnership model… • Allows access to the entire
historical archive of large datasets
• Uses a market to provide choices to industry without NOAA interference
• Increases the reach of NOAA data
• Frees up NOAA personnel from repeating data extracts
![Page 9: The NOAA Big Data Project Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062302/58ec83df1a28ab666f8b45ad/html5/thumbnails/9.jpg)
Reducing Operational Time at The Climate Corporation
![Page 10: The NOAA Big Data Project Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062302/58ec83df1a28ab666f8b45ad/html5/thumbnails/10.jpg)
Reducing Operational Time at The Climate Corporation
![Page 11: The NOAA Big Data Project Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062302/58ec83df1a28ab666f8b45ad/html5/thumbnails/11.jpg)
Reducing Operational Time at The Climate Corporation
• Project timelines at TCC are now several weeks shorter
• TCC can evaluate new forecasting models on larger datasets
• TCC reduces project cost because they only pay for temporary cloud storage and compute cycles
• NOAA improves archive through resolution of data issues
![Page 12: The NOAA Big Data Project Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062302/58ec83df1a28ab666f8b45ad/html5/thumbnails/12.jpg)
Types of NOAA Data
![Page 13: The NOAA Big Data Project Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062302/58ec83df1a28ab666f8b45ad/html5/thumbnails/13.jpg)
NEXRAD Doppler Radar
![Page 14: The NOAA Big Data Project Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062302/58ec83df1a28ab666f8b45ad/html5/thumbnails/14.jpg)
NEXRAD Doppler Radar
• Original resolution (Level II) available now
• Precipitation and atmospheric movement (wind)
• Tornado visualization
• Cold fronts
• Thunderstorm gusts
• Rainfall rate/hydrological forecasting
![Page 15: The NOAA Big Data Project Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062302/58ec83df1a28ab666f8b45ad/html5/thumbnails/15.jpg)
VIIRS Infrared Satellite
• Covers the entire globe every 14 hours
• Enables short-term monitoring of:
• Wildfires
• Drought
• Vegetation
• Ocean temperature
![Page 16: The NOAA Big Data Project Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062302/58ec83df1a28ab666f8b45ad/html5/thumbnails/16.jpg)
VIIRS Infrared Satellite
• Assists in long-term assessment of:
• Emerging global storm patterns
• Global economic activity
• Arctic and ocean ice
![Page 17: The NOAA Big Data Project Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062302/58ec83df1a28ab666f8b45ad/html5/thumbnails/17.jpg)
IOOS Ocean Observation
• 10,000+ oceanographic data sets
• Buoys
• High-frequency radar
• Water level gauges
• Gliders
• Animal telemetry
• Coastal and estuary stations
![Page 18: The NOAA Big Data Project Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062302/58ec83df1a28ab666f8b45ad/html5/thumbnails/18.jpg)
Questions?
![Page 19: The NOAA Big Data Project Overview](https://reader036.vdocuments.mx/reader036/viewer/2022062302/58ec83df1a28ab666f8b45ad/html5/thumbnails/19.jpg)
Contact Information
Email: [email protected]
Twitter: @AmyVGaskins
LinkedIn: www.linkedin.com/in/amygaskins