cloud hosting water science data for collaboration · 2018-04-20 · what is cuahsi? •cuahsi is a...
TRANSCRIPT
Cloud Hosting Water Science Data for CollaborationMARK HENDERSON
CUAHSI SENIOR ENGINEER
WWW.CUAHSI.ORG
Outline
WWW.CUAHSI.ORG
Recap: About CUAHSIRecap: Intro to Data ManagementWhat HydroShare Is
OverviewDiscover Demo: Discovering HydroShare Resources
The Steven Hall StoryCreating HydroShare Resources
How to store data in the cloud to encourage collaborationDemo: Uploading a resourceDemo: Extended MetadataDemo: Publishing + DOIDemo: Access and Groups
Preview: Reproducible science
What is CUAHSI? • CUAHSI is a 501(c)3 Non-Profit Consortium of about 130 U.S. Academic Institutions, Non-
Profits, and International Universities; also open to private organizations
CUAHSI's mission is to shape the future of hydrologic science by:◦ Strengthening collaboration
◦ Developing and delivering data, models, instrumentation and technologies
◦ Promoting education
• Key Activities
• Community Services, such as workshops, community meetings, training, etc.
• Data and Model Services, including HydroShare and time-series services
Data Management Lifecycle Revisited
WWW.CUAHSI.ORG
How do people share other content now?
oYouTube
oFacebook
oInstagram
oDrop Box
oGoogle Drive
oArcGIS Online
oHydrologic data ?
Slides by David Tarboton
Data and models used by hydrologists are diverse…
Time series
Geographic rasters
Geographic features
Multidimensional space/time
Model programs
Model instances
…
Slides by David Tarboton
141 241 341
131 231 331
121 221 321
111 211 311
441
431
421
411
142 242 342
132 232 332
122 222 322
112 212 312
442
432
422
412
143 243 343
133 233 333
123 223 323
113 213 313
443
433
423
413
Y
X
Time
http://www.unidata.ucar.edu
http://www.usgs.gov
http://www.esri.com
From Jeff Horsburgh
HydroShare is a system for collaborating around, sharing and publishing Hydrologic data, models, and code
oFile Storage
oMetadata Descriptions
oData Access API
oWeb Apps
oSocial Functions
oDOI Data Publication
Slides by David Tarboton
Value Added Functionality
DropBox-ish Functionality
dropbox.com
What is a HydroShare “Resource?”
WWW.CUAHSI.ORG
AKA “items” or “Projects.” it is a value-added collection of data andmetadata.
For each resource you can:o Manage who has accesso Get unique identifiero Describe with metadata (Dublin Core Metadata Standard)o Organize into collections/groupso Formally publish (DOI)o Versiono Open with compatible web app
Slides by David Tarboton
Types of Data AvailableResource types help to:
◦ Provide resource-specific metadata (file-type specific)
◦ Provide different content organization
◦ Provides grouping/hierarchical structure of different data
WWW.CUAHSI.ORG
General• Composite• Generic• Collection• Web App
Slides by David Tarboton
Geographic• Geographic Feature (ESRI Shapefile)• Geographic Raster• Multidimensional (NetCDF)
Modeling• Model Program• Model Instance• SWAT Model Instance• MODFLOW Model
Instance• Script
Time Series• Time Series
Demonstration
◦ Discover – search for other data, models, and code
◦ Collaborate – find a group of researchers sharing data related to a project or
theme or create a group to share data privately with a few colleagues
◦ Apps – Use tools that operate on your data
◦ Help – Get information on anything HydroShare-related
◦ My Resources – my library
◦ Upload example
WWW.CUAHSI.ORG
Data Discovery in HydroShareMultiple Types of Searching
◦ Free Text with keywords/labels◦ Facets for easy filtering◦ List and Map Views
WWW.CUAHSI.ORG
Demo!
Creating a ResourceCreating a publishable resource in HydroShare is simple:
At a minimum, a resource requires:
1. Title
2. Content Files
3. Abstract
4. Keywords
WWW.CUAHSI.ORG
Demo!
Automatic and natural metadata gathering eases some of the pain of metadata entry
WGS 84 Coverage information is automatically harvested from GeoTIFF coordinate system information
For multidimensional netCDF data with CF convention metadata it is automatically completed on HydroShare
Get your resource ready for publication!◦ Fill out metadata including keyword tags
◦ Manage access if necessary
◦ Choose the right license for your data
◦ Give your funding agency credit
◦ Retrieve your DOI pattern to put in your associated paper publication◦ http://doi.org/10.4211/hs.
◦ https://www.hydroshare.org/resource/a3a9ba772aac4cba9533b35bb6b5fe42/
◦ http://doi.org/10.4211/hs. a3a9ba772aac4cba9533b35bb6b5fe42/
◦ Once the paper is published, add the paper DOI to the resource
◦ Agree to the HydroShare terms of use and submit for publication
WWW.CUAHSI.ORG
Maintain Metadata (Information about Data)oWHO created the data?
oWHAT is the content of the data?
oWHEN were the data created?
oWHERE is it geographically?
oWHY were the data developed?
oHOW were the data developed?
WWW.CUAHSI.ORG
Borer et al.: “Do not underestimate your ability to forget details about a study!” Demo!
Managing Access to Data and ModelsShare a resource privately to:◦ Another HydroShare user◦ A group on HydroShare
Share a resource to public:◦ “Public”◦ Others can discover and download the resource on the HydroShare Discover page
Share a resource to the public but restrict access:◦ “Discoverable”◦ Others can discover the resource metadata but cannot download the data
Allow others to discover and share your resource with others◦ “Shareable”
WWW.CUAHSI.ORG
Demo!
Formal Data Publication
WWW.CUAHSI.ORG
Resources formally published receive a citable digital object identifier (DOI) and are made immutable to changes
Demo!
WWW.CUAHSI.ORG
Summary
o A new, web-based system for advancing model and data sharing
o Access multiple types of hydrologic data using standards compliant data formats and interfaces
o Flexible discovery functionality
o Model sharing and execution
o Facilitate and ease access to use of high performance computing
o Social media and collaboration functionality
o Links to other data and modeling systems
o Enable more rapid advances in hydrologic understanding through collaborative data sharing, analysis and modeling
o Much of the functionality has applicability to other geosciences beyond hydrology
CUAHSI’s Programs
Community Services
Programs that build capacity in specialized topics and provide transformative learning experiences.
Data Services
Infrastructure for sustainable archiving, sharing, analyzing, and modeling scientific data and developing new products that are useful to the water science community.
WWW.CUAHSI.ORG
Community Services
◦ Learn a new instrumentation method
◦ Engage your local community on water issues
◦ Develop or improve data tools
◦ Add a new field site to existing research
◦ Collaborate with researchers at other institutions
WWW.CUAHSI.ORG
CUAHSI provides access to resources and specialized training to build capacity and extend existing research.
Cyberseminars(At least) Fall and Spring Series each year
Over 150 recorded webinars on specialized water topics
Upcoming Topics
April 13: Data Archiving and Dissemination Tools to Support Water Science Research
April 20: Cloud-hosting Water Science Data for Collaborative Research
April 27: Cyberinfrastructure to Support Water Science Education and Reproducible Science
WWW.CUAHSI.ORG
Trainings and WorkshopsOne of the best ways to master a new method or instrumentation technique is immersion in the field with experts in the technology.
We offer training on specific technologies or theories that may not be available through any one institution.
April 2018: Field Experiences in Microwave Remote Sensing for Agricultural Hydrology
June 2018: The Community WRF-Hydro Modeling System
September 2018: Near Surface Geophysics for Hydrology
WWW.CUAHSI.ORG
Pathfinder Fellowship
Go beyond "one site, one view" in your graduate research.
Graduate students are eligible.
Key dates:◦ Proposals due October
◦ Awards made December
◦ Travel occurs January-December
WWW.CUAHSI.ORG
Instrumentation Discovery Travel GrantsLearn the details of hydrologic instrument installation, operation, maintenance, and data processing by visiting experts or scheduling reverse site visits. Graduate students and faculty are eligible.
Key Dates◦ Proposal Deadline September 30
◦ Awards made by Nov 1
◦ Proposal Deadline April 30
◦ Awards made by June 1
WWW.CUAHSI.ORG
National Water Center Innovators Program: Summer Institute
Partnership between CUAHSI and the National Weather Service to help engage the academic research community in the enhancement of the National Water Model.
Seven-week, onsite rapid prototyping of ideas
Graduate students are eligible.
Key Dates◦ November 1: Applications open
◦ December 18: Deadline to submit applications
◦ Mid-January: Awards issued
◦ June 12: Registration and orientation at the University of Alabama
◦ July 24: Summer Institute Capstone Meeting
WWW.CUAHSI.ORG
Community Meetings
Hydrologic Connections: Climate, Food, Energy, Environment, and Society
July 29 – August 2 at NCTC
Shepherdstown, WV
Keynote Speakers:• Kristen Averyt (Desert Research
Institute)
• Larry Band (University of Virginia)
• George Hornberger (Vanderbilt University)
www.CUAHSI.org
Save the Date!
2018 CUAHSI Biennial Colloquium
Data Services
WWW.CUAHSI.ORG
◦ Share, collaborate, discover, analyze, model, and publish all types of water data
◦ Free, open-source software
CUAHSI provides access to data and models to facilitate research and education.
◦ Fulfill data management plans
◦ Use for research or classroom
◦ Develop new software to meet community needs
Demo!
Demo!
Sharing Data
Providing data: ◦ Why were the data created?
◦ What limitations do the data have?
◦ What does the data mean?
◦ How should the data be cited if it is re-used in a new study?
Receiving data:◦ What are the data gaps?
◦ What processes were used for creating the data?
◦ Are there any fees associated with the data?
◦ In what scale were the data created?
◦ What software do I need in order to read the data?
◦ What projection are the data in?
◦ Can I give these data to someone else?
WWW.CUAHSI.ORG
Jupyter Python Notebook App
Write and execute python code acting on content of HydroShareresources and saving results back to HydroShare
Thank you!
Contact Information
Mark Henderson
Senior Software Engineer
339-226-7445
https://www.hydroshare.org/resource/184622da559d400cb7edd5f39a302b04/
WWW.CUAHSI.ORG