australian bureau of meteorology water information program - building a national water database tony...
TRANSCRIPT
Australian Bureau of Meteorology Australian Bureau of Meteorology Water Information Program - Water Information Program -
Building a national water databaseBuilding a national water database
Tony BostonTony Boston
Assistant DirectorAssistant Director
Water Data ServicesWater Data Services
OutlineOutline
• Bureau of Meteorology - Water Information Bureau of Meteorology - Water Information ProgramProgram
• Water Act 2007Water Act 2007 and Regulations and Regulations• Water data transfer standardsWater data transfer standards• Ingestion of water observations dataIngestion of water observations data• Australian Water Resources Information Australian Water Resources Information System (AWRIS)System (AWRIS)
• Australian Hydrological Geospatial Fabric Australian Hydrological Geospatial Fabric (Geofabric)(Geofabric)
Bureau of Meteorology functions
National Weather Service
National Weather Service
National Tsunami Alert Service
National Tsunami Alert ServiceNational Tidal CentreNational Tidal Centre
National Flood Warning and
Forecasting Service
National Flood Warning and
Forecasting Service
National Climate Monitoring SystemNational Climate
Monitoring System
National Ocean Current PredictionNational Ocean
Current Prediction
National Water Information Service
National Water Information Service
Climate and Meteorological
Research(with CSIRO)
Climate and Meteorological
Research(with CSIRO)
Rainfall Deciles – 1 November 2001 to 31 October 2008
Distribution based on Gridded Data Product of the National Climate Centre
The water information value ladder
Measurement
Quality assurance
Archiving in house
Integration
Analysis
Reporting
Forecasting
Distribution
Rarely done
Generally done poorly
Generally done well, by over 100 groups, butcould be vastly improved with new
technology
>>> Incre
asing value >>>
Data >>> Info
rmatio
n >>> Insight
Bureau of Meteorology - Water Information ProgramBureau of Meteorology - Water Information Program
• 10-year Commonwealth program, started July 07
• $450m funding
– 110 new staff + systems + R&D
– support for data providers
• Legislative backing
– Water Act 2007
– Water Regulations 2008
• Relies on collaboration with data providers
The Bureau will become Australia’s The Bureau will become Australia’s leading water information providerleading water information provider
Provider dataProvider dataStreamflow
Groundwater
Water quality
Water use
Entitlementsand Trades
Storage
Diversions
Various spatialdata layers
AWRISAWRIS
Water Data
Geofabric
Hydro DB
Climate DB
Informationproducts
Informationproducts
REPORTING SERVICES
FORECASTINGSERVICES
Browser, PDA, RSS, XML
Dynamic
NATIONAL WATER
RESOURCE ASSESSMENT
NATIONAL WATER
ACCOUNT
Rolling annual reports
Static
Water Regulations 2008Water Regulations 2008
• Specify a complex matrix of categories of data, types of Specify a complex matrix of categories of data, types of organisation and timeframes for delivery to the Bureauorganisation and timeframes for delivery to the Bureau
• Initial Regulations ask for any data in electronic formatInitial Regulations ask for any data in electronic format
=> lots of ‘CSV’ with no schema!=> lots of ‘CSV’ with no schema!
• Bureau can define National Water Information StandardsBureau can define National Water Information Standards
• Data transfer standards will be mandated in future versions Data transfer standards will be mandated in future versions of the Water Regulationsof the Water Regulations
Categories of water informationCategories of water information
1. surface water resources
2. groundwater resources
3. information on major and minor storages
4. meteorological information
5. water use
6. rights, allocations and trades
7. urban water management
8. water restrictions
9. water quality
10.metadata for the above
Types of OrganisationTypes of Organisation
A. Lead State/Commonwealth water agencies
B. Other State/Commonwealth agencies
C. Hydroelectricity generators
D. Owners or operators of major storages
E. Rural water utilities
F. Urban water utilities
G. CMAs and like organisations
H. Flood level information providers
Data Transfer to Bureau of Meteorology
Canberra
Source: David Maidment 2009
Water Regulations data delivery
• ten primary categories of data (~65 variables)
• eight categories of person (246 persons in total)
• 104-105 observation points with time series
• entire historical archive provided at first
• updated thereafter real time, daily, weekly, monthly or yearly
Oct 08 Feb 09 Apr 09 Jul 09
Water data transfer standardsWater data transfer standards
• Under WIRADA research alliance, CSIRO and the Bureau Under WIRADA research alliance, CSIRO and the Bureau have developed the Water Data Transfer Format (WDTF)have developed the Water Data Transfer Format (WDTF)– Version 0.2 September 2008Version 0.2 September 2008– Version 0.3 February 2009Version 0.3 February 2009– Version 1.0 October 2009Version 1.0 October 2009
• Developed through the Bureau’s Water Information Developed through the Bureau’s Water Information Research and Development Alliance (WIRADA) with CSIROResearch and Development Alliance (WIRADA) with CSIRO– $50M over 5 years$50M over 5 years
Water Data Transfer Format (WDTF)Water Data Transfer Format (WDTF)
• XML format XML format
• Based on standards stack from Open Geospatial Based on standards stack from Open Geospatial Consortium (OGC):Consortium (OGC):– Geography Markup Language (GML)Geography Markup Language (GML)– Observations and Measurements (O&M)Observations and Measurements (O&M)– Simple features profileSimple features profile
=> compatible with WFS and SOS=> compatible with WFS and SOS
=> compatible with other environmental data => compatible with other environmental data
Development of WaterML 2.0
• Aim: –Harmonize Australian WDTF and CUAHSI WaterML 1.X –Link to more hydrologic feature-types . not just point monitoring sites
– Improved vocabulary/ontology management, mapping, publication
international standard for water observations data, interoperable with other domains in environmental sciences–CSIRO–CUAHSI–OGC Hydrology Domain Working Group–WMO Commission for Hydrology – BoM
WaterML 2.0 evolution
AWRIS Core Solution OverviewAWRIS Core Solution Overview
Systems Management
Data Ingest Data Maintenance
Data Storageand Versioning
File Package Acquisition
(FTP, CD, Email)
Audit, Logging and Notification
Messaging Access Control(Identity Mgmt, Security)
Process Configuration
(Rules and Dependencies)
Operations Support DBA and Archiving
Provider Feedback
Data Return
External Data Exchange
(Flood, ADAM, AWDIP)
File Storeand Catalogue
Format Standardisation
Normalisation and Validation
Holdings Catalogue
Data Export
Data Aggregation Data Derivation
Manual DataEntry / Edit
Reference Data(Spatial Data Sets,
Identifier Management)
Product Publishing
Content Management
ContentGeneration
Dashboards
Mapping Visualisation
Search Download
Publishing
AWRIS: Receive Regulations DataAWRIS: Receive Regulations Data
File statisticsFile statistics
FileType TotalFiles TotalMBUnknown 24 4.36generic text 995 5.04generic CSV 15,169 2,919.48generic Excel 384 114.78generic XML 40,280 988.06generic PDF 72 38.33generic TIF 1 0.07generic DXF 2 1.50generic DOC 9 0.73Wiski export 29,657 1,624.80Hydstra export 1,328,558 24,032.89WDTF 298,659 25,153.69TOTAL 1,713,810 54,883.71
The Australian Hydrological Geospatial Fabric (the Geofabric)
The Geofabric will be a suite of authoritative spatial data products forming a consistent, national geospatial framework for hydrological features. It will contain a consistent representation of water features and their connectivity in the Australian Water System.
The Geofabric will become the geospatial information framework for Australia’s water information activities.
AWRIS: Present web productsAWRIS: Present web products
Web User
AWRIS products
• Find–Map interface and text searching
• Get–Data download, plots, dashboards, maps
• Web services for SpatioTemporalKeyword searching–WFS or SOS from holdings catalogue–WDTF or WaterML 2.0 for time series observations data
• Customisation–MyAWRIS and mobile applications
• Static content–National Water Accounts–Water Resource Assessments
Australian Dam Storage and Level Information (ADSLI)
Australian Dam Storage and Level Information (ADSLI)
Australian Dam Storage and Level Information (ADSLI)
Australian Dam Storage and Level Information (ADSLI)
Licensing of data and information productsLicensing of data and information products
• Australian governments have traditionally applied Crown Australian governments have traditionally applied Crown Copyright: Copyright: ©Commonwealth of Australia©Commonwealth of Australia
• Movement towards open access to public sector informationMovement towards open access to public sector information– National Government Information Licensing FrameworkNational Government Information Licensing Framework– Built on Creative Commons (mainly Attribution) licensingBuilt on Creative Commons (mainly Attribution) licensing
• The Bureau would like to use Creative Commons for water The Bureau would like to use Creative Commons for water data and information productsdata and information products– Requires agreement from data suppliersRequires agreement from data suppliers– Phased implementationPhased implementation
