u.s. department of the interior u.s. geological survey data integration progress and guiding...
TRANSCRIPT
U.S. Department of the InteriorU.S. Geological Survey
Data Integration Progress and Guiding Principles
Disciplines, generalization, and open-access.
David Blodgett – [email protected] Office of Water Information Center for Integrated Data Analytics
Outline
· Data Integration Disambiguation
· Barriers to moving Forward.
· Anecdotes, everyone loves anecdotes!
· Principles to go Forward!
Disclosures
· I’m a water guy.
· I‘m a millennial.
· I assume Internet.
· I’m a Badger.· … Forward!
Data Integration – Disambiguated.
Integration is the act of combining multiple things into a whole.
Data Integration – Disambiguated.
What makes something integrated?
How different do things need to be to count?
Do you just need to combine things?
What kind of data integration is needed for decisions?
2014-
05-
12
7
Visual Integration
Data Consolidation
DataWarehouse
Data Bundling Data Fusion
Integrated Search Multi-source Data Ingest
in the Cloud?
Application / Decision Driven Model of Data Integration Slide Credit: Jeff de La Beaujardiere
What kind of data integration is needed for decisions?
2014-
05-
12
8
Visual Integration
Data Consolidation
DataWarehouse
Data Bundling Data Fusion
Integrated Search Multi-source Data Ingest
in the Cloud?
A general model for data integration.
Disciplinary Details
Free and Open Service
Access
Generalized Standards
Service Orientation
On local machines, we run software.List, introspect, summarize, transform, integrate.
Can scan the entire domain of the data!
A service may do any or all of these things.
Software on the server can summarize the domain and range of its holdings.
(ie. Deliver Dynamic Metadata)
Web Service – So what?
Software on the server
can summarize the
domain and range of its
holdings.
Generalized Aspects of Data Services
Spatial/ Temporal
Extent
Attribute Extent
Blob of Bits
Available Formats
International Standards.
Various Communities’ Interchange
Discipline specific linked to other disciplines.
Practical Barriers
‘I don’t know how to use the required software.’
‘The software I need is really expensive.’
‘The information I need is a big mess.’
‘The information I need is really big.’
Understanding Barriers
‘The information is in a language I don’t know.’
‘The information is in a format I’ve never seen.’
‘The taxonomy used doesn’t work with mine.’
‘I’m not sure if what I’m seeing is a data quality issue or real.’
Defensive Barriers
‘I collected this data and want to publish on it.’
‘People won’t interpret my data correctly.’
‘I don’t want to be liable for decisions made.’
‘This data’s quality is too low to stand behind.’
Square Pegs and Round Holes
Coverages and Features
A grid cell IS NOT a point measurement!!!
Scale Discontinuity
Anecdotes!...
Because they are instructive!
Water Quality Portal
http://www.waterqualitydata.us
USGS, EPA, USDA Joint service providing water quality and other environmental monitoring data.
Integrated Ocean Observing System
Weather Underground42K Current Conditions Weather Stations!
WeatherCommon architecture for access and processing multiple environmental data resources!
Geo Data Portal Data Integration Framework
Center for Integrated Data Analytics: Nate Booth, Tom Kunicki, Dave Blodgett, Jordan Walker, Ivan Suftin, I-Lin Kuo.
Landscape
Climate
Enabling Technologies….
____.data.gov – Big Win!
Data access type is a first class citizen!
Includes both human and machine metadata.
Machine-interpretability is an expectation.
Content management systems and catalogs are becoming data service providers!!!
Forward!
Principle #1: Data Object Patterns
We must continue to identify and model the common patterns our data adhere to.
Non-interpretive content / attributes should be provided by service ‘methods’.
These patterns must transcend discipline or implementation.
Principle #2: Domain Semantics.
Semantic relationships are necessarily governed by a given scientific domain itself.
This is Foundational to all additional interdisciplinary concerns.
Principle #3: __ - Agnostic Standards
Standards, specifications, and best practices must be ____ - agnostic.
A standard can be implemented using any technology, in any discipline.
eg. WaterML2 -> TimeSeriesML
Principle #4: Identity Management
Uniqueness can’t be taken for granted and must be curated very deliberately.
You are not your location. Neither is a place.
Foundational to linking any and all information to an entity.
A few thoughts to leave you with…
Maps are metadata.
Index-based data access is dead.
A Geospatial database should be coherent without it’s spatial table.
Summary
A standard is an established generalization.
Scientific disciplines govern their semantics.
Open-access (the internet) must be a given.