gsics data management working group
DESCRIPTION
GSICS Data Management Working Group. 15 th Session of Executive Panel Guangzhou, China. Manik Bali(Chair), Aleksandar Jelenak , Rob Robelling , Peter Miu , Masaya Takahashi, Byung -II NOAA Satellite and Information Service. 16 May 2014. Outline. Introduction - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/1.jpg)
GSICS Data Management Working Group
Manik Bali(Chair), Aleksandar Jelenak,
Rob Robelling, Peter Miu, Masaya Takahashi, Byung-II NOAA Satellite and Information Service
15th Session of Executive Panel Guangzhou, China
16 May 2014
![Page 2: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/2.jpg)
Outline
• Introduction
• GDWG Membership News.
• GDWG –Progress in Action Items .
• GSICS Communications
• GSICS user messaging service.
• GSICS Dev server.
• Review of activities since the last meeting.
• NOAA. JMA, EUMETSAT
• CMA, KMA
• Roadmap-2014
• Summary/Conclusions.
2
ProgressPast 1 Yr
Future
15th Session of Executive Panel Guangzhou, China16 May 2014
![Page 3: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/3.jpg)
• Min Min• Zhigou Rong• Yuan Li • Peter Miu• Simon Elliott• R. K Giri
3
• Masaya Takahashi• Younghooi Hwang• Sergey Uspensky• Kazuo Umezawa• Jerome Lafeuille
Introduction
GDWG Membership News
• Masaya Takahashi has indicated that he would become the chair when he gets back to JMA.
• Peter Miu has indicated that he would be the vice chair.15th Session of Executive Panel Guangzhou, China16 May 2014
![Page 4: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/4.jpg)
• 26 action items from the last Annual Meeting.
• 24 completed.
• 2 overdue (some reflect change of circumstances; some will likely be
reformulated at this meeting)
4
Review of activities since the last meeting
Action Items Progress
15th Session of Executive Panel Guangzhou, China16 May 2014
![Page 5: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/5.jpg)
5
GSICS User Messaging Service
• 259 Subscribers.
• Since last annual meeting 41 new subscribers added.
Communication
Registration in GUMS is steadily increasing
GDWG maintains the GUMS
May-13 Jun-13 Jul-13 Aug-13 Sep-13 Oct-13 Nov-13 Dec-13 Jan-13 Feb-13 Mar-13 Apr-13200
210
220
230
240
250
260
270Registrations in GUMS
Registrations in GUMS
15th Session of Executive Panel Guangzhou, China16 May 2014
![Page 6: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/6.jpg)
6
Tim HewisonFred Wu
BobAleksandar
PeteManik
Jerome
0 50 100 150 200 250 300 350
297
47
34
29
21
20
18
Contributors to GSICS Forum
Number of all time posts
GSICS Dev Server
Communication
GDWG maintains the Dev Server
![Page 7: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/7.jpg)
7
Review of activities since the last EP meeting
IMD GDWG progress
• GEO and LEO Data receiving stations have been setup.
• Real time visualization and quality monitoring for NPP, METOP-A/B , MODI NOAA available
• Infrastructure for cross calibration being setup.
15th Session of Executive Panel Guangzhou, China16 May 2014
![Page 8: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/8.jpg)
• Support to GCC for the transition of MW FCDR product demo and then to pre-
operational phase.
• SRF of GEO instruments in NetCDF format [support to DCC]
• Developed Product Taxonomy This is supposed to be the basis for redoing the THREDDS configuration to group GSICS correction products differently. Also the taxonomy would
come handy in planning how to utilize the file naming convention and any other data management task.
• Developed a data model for satellite instrument event logs and translated it into an
extension to the ISO 19115-2 metadata standard's XML Schema.
8
Review of activities since the last meeting
NOAA GDWG progress
15th Session of Executive Panel Guangzhou, China16 May 2014
![Page 9: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/9.jpg)
• Initiated process to get DOI number for all GSICS products and documents. DOI for MW already assigned by NOAA ( outside of GSICS ) GDWG to have a web meeting in June 2014 to decide on doi Vs oid.
• Other GSICS products need review to progress to op.
(need clarity from EP on archival & GPPA modification, Chair’s decision)
• New website for GDWG ( content requested from members).
• New GPPA ( proposed by Peter Miu, Manik Bali).
9
Review of activities since the last meeting
NOAA GDWG progress
15th Session of Executive Panel Guangzhou, China16 May 2014
![Page 10: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/10.jpg)
JMA GDWG progress
10
• Satellite event log
• Provided the information on the JMA website in Mar/June 2013
• Change of the GEO-LEO IR RAC NetCDF contents
• Met the requirements of EUMETSAT bias plotting tool in July 2013
• Provision of official SRF data to GRWG VIS/NIR sub-group
• Provided data to GDWG in Apr 2013
• Collaboration with WIS
• Reported feasibility to the GDWG chair in May 2013
• Calibration change alerts
• Reported feasibility to gsics-dev in June 2013
Review of activities since the last meeting
15th Session of Executive Panel Guangzhou, China16 May 2014
![Page 11: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/11.jpg)
11
• Proposed New GPPA.
• Support to all GDWG activities.
• Visualization tool.
EUMETSAT GDWG progress
Review of activities since the last meeting
15th Session of Executive Panel Guangzhou, China16 May 2014
![Page 12: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/12.jpg)
12
•Before leaving GSICS, the NOAA GDWG member took a lead in the development of a model for a common satellite event logging system.
•The model has been briefly evaluated by EUMETSAT and their feedback it that it is too complex for the needs of GSICS. Further evaluation needs to be done.
•EUMETSAT is submitting a CGMS paper proposing a way forward to simplify the approach to the resolution of this action.
•Based on the response to this paper, the appropriate actions will be taken.
EUMETSAT (Miu)
CGMS is tasking GSICS to support activities relevant to their actions (e.g. Event logging). If progress is expected in this or any GSICS action, the GSICS EP needs to urgently and decisively take a lead in managing this long standing issue
EUMETSAT GDWG progress
On Event Logging Data Model…..
16 May 201415th Session of Executive Panel Guangzhou, China
![Page 13: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/13.jpg)
EUMETSAT on Bias Monitoring tool….• The tool is under development.
13
EUMETSAT GDWG progress
15th Session of Executive Panel Guangzhou, China16 May 2014
![Page 14: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/14.jpg)
14
CMA progress
• IASI acquirement transferred from CLASS to EUMETSAT
pushing (March, 2013)
• Update GSICS data collection system for FY new sensors
(FY-3C) and new reference instrument (NPP-CrIS, NPP-
VIIRS)
• Automatically PICS data extraction for FY-3/MERSI/VIRR
• Provide the instrument telemetry/Engineering data (OBC
files) from FY operational processing system
• CMA GPRC new website is developing and will finish
shortly
Review of activities since the last meeting
15th Session of Executive Panel Guangzhou, China16 May 2014
![Page 15: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/15.jpg)
15
KMA progress
Review of activities since the last meeting
• Developing KMA’s GSICS website
• General Introduction of COMS inter-calibration
• Methods and Bias Monitoring for IR and VIS
• Links to Other GSICS Website
• Corrected Data filename and format
15th Session of Executive Panel Guangzhou, China16 May 2014
![Page 16: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/16.jpg)
16
KMA GSICS
Internal Data Collection
NOAA NASA
External Data Collection
COMS Data MODIS, OMI, AIRS, IASI
CloudVISCC
DesertVISCD
OceanVISCO
IRAIRS/IASI
NetCDFModule
StatisticsModule
Image ModuleData
Conversion
NMSC Homepage
DataProcess
DataCollection
DataDisplay
Four steps
1) Data Collection
• COMS, IASI, AIRS, MODIS, OMI
2) Data Processing
• Daily, Monthly Process and archiving
3) Data Conversion
• NetCDF format conversion
4) Display
• GSICS in NMSC Website
• Visible and Infrared
Storage Storage
KMA, Byung
KMA progress
16 May 2014
![Page 17: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/17.jpg)
17
MT1R_20130531_0600_AIRS_060.nc MT1R_20130531_1100_IASI_1008449763.IASI_xxx_1C_M02_20130531112 958Z_20130531113254Z_N_O_20130531125921Z__20130531130029.nc
AIRS : W_KR-KMA-NMSC,SATCAL+NRTC+GEOLEOIR,COMS+MI-
Aqua+AIRS_C_RKSL_201305310600.nc
IASI : W_KR-KMA-NMSC,SATCAL+NRTC+GEOLEOIR,COMS+MI_
MetopA+IASI_C_RKSL_201305311100.nc
Old Filename
From April 1, 2011 to present
New Filename : GSICS standard
pflag_productidentifier_oflag_originator_yyyyMMddhhmmss[_freeformat].type[.compression]
KMA progressKMA GSICS
KMA, Byung15th Session of Executive Panel Guangzhou, China16 May 2014
![Page 18: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/18.jpg)
FROM RESEARCH TO PRODUCTIONBASIC PATHWAYS/TOOLS
HAVE BEEN MET
1815th Session of Executive Panel Guangzhou, China16 May 2014
![Page 19: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/19.jpg)
19
• Maintenance of current services in 24x7 mode
• THREDDS Server
• Product Catalog and Products archival.
• Wiki Page/ Web Page
• GDWG Support to GPPA to enable acceptance of Newer Data into
GSICS fold.
• Centralized Bias monitoring system.
Road Map for 2014
15th Session of Executive Panel Guangzhou, China16 May 2014
![Page 20: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/20.jpg)
• Satellite Instrument Event Logging specification, design and implementation.
• Identify a process for archiving GSICS products- need a policy document
• Implementing new capabilities in the GSICS product plotting tool.
• GSICS document management plan definition and associated document templates.
• Doi /Oid numbers for Documents and Products.
• Address resource crunch
20
Road Map for 2014
15th Session of Executive Panel Guangzhou, China16 May 2014
![Page 21: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/21.jpg)
• Perform analytics when instruments are compared with
stable references
• Detect instrument degradation trends
• Detect impacts of diurnal/seasonal instrument
temp variations.
21
Road Map for 2014
Asses New tools like Big Data that can
15th Session of Executive Panel Guangzhou, China16 May 2014
![Page 22: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/22.jpg)
• NOAA GDWG has been able to ensure running of key services like Product
Catalog, GSICS Dev and GUMS
• File naming and DOI number for products.
NOAA has the resources to assign DOI number .
Outside of GSICS MW product already got the doi number.
22
Summary
15th Session of Executive Panel Guangzhou, China16 May 2014
![Page 23: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/23.jpg)
• Progress Reports NOAA, CMA, JMA , KMA. Excellent progress demonstrated by members towards achieving
goals of GSICS.• Calibration Change Alerts ( Peter Miu, Rob Robelling)
Automation, Connection , Harmonization and online depiction discussed.
• Big Data (Lauri Rokke) Performance of algorithms, analytics
• GSICS Archiving Process ( Peter Miu) Procedure for archiving, where to archive and until when.
23
Summary
15th Session of Executive Panel Guangzhou, China16 May 2014
![Page 24: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/24.jpg)
• Event Log ( Rob). WMO to include GPRC event logging links to the OSCAR website.
• GPPA improvements ( Peter Miu, Manik Bali ) To make GPPA submission online and reduce time of review.
• GEO-LEO Product promotion ( Manik Bali ). GDWG to extend file naming and meta data standards to other
channels. • GDWG to support to User workshop 2014 .
24
Summary
15th Session of Executive Panel Guangzhou, China16 May 2014
![Page 25: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/25.jpg)
25
Successfully collaboration requires all GSICS GPRCs to:1. Nominate a representative to the GDWG.2. Each GDWG representative needs to commit time to work
on GSICS activities (see details in the 2014 GDWG minutes).
CGMS is tasking GSICS to support activities relevant to their actions (e.g. Event logging). If progress is expected in this or any GSICS action, the GSICS EP needs to urgently and decisively take a lead in managing this long standing issue.
Summary
• Harmonize working hours of GDWG members.
![Page 26: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/26.jpg)
26
THANKYOU
15th Session of Executive Panel Guangzhou, China16 May 2014
![Page 27: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/27.jpg)
GDWG status of members.The following GPRCs are represented in the GDWG:• EUMETSAT – 2 members• JMA – 1 member• KMA – 1 member• NOAA – 1 member
Successfully collaboration requires all GSICS GPRCs to:1. Nominate a representative to the GDWG.2. Each GDWG representative needs to commit time to work on
GSICS activities (see details in the 2014 GDWG minutes).
. 27
![Page 28: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/28.jpg)
28
Advise the Executive Panel in the planning and
implementation of data management activities.
IntroductionGSICS EP
GPRC GCC
GCC Dir.
GCC Deputy IR UV VIS MW
![Page 29: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/29.jpg)
2008• Concept of Catalogue.
• Data format. • Event log.
• Data Management Services
5
2007• Archiving requirements. • Access methods.
• Data sets.
• Meta Data v Documentation v Security Requirements.
2009
• GPPA (Product Acceptance )
• Docs on WIKI
• NetCDF file definitions
2010• The essential requirements of the server as a
data exchange platform were satisfied. The
server was ready for operations.
2011
• Bias Monitoring.
• NetCDF file naming conventions
2012
• GSICS User Messaging Service
• NetCDF file naming conventions
• THREDDS Config. finalization2013
• Product Catalogue.
• Web-based bias monitoring tool
2014
Concept Implementation
Introduction Milestones since 2007
![Page 30: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/30.jpg)
Key Discussions and Action items• Resources needed to maintain the running of servers, websites
,catalogs• AK Sharma from IMD indicated that IMD would identify a
person to work on GDWG.• Accounts on Wiki would be provided to Masaya and Peter.• Masaya (VC), Peter ( Nominated himself VC).• Harmonize working hours of GDWG members.• Aleksandar’s ideas on Taxonomy for products was presented.• It was felt that currently we don’t have resources to
implement it at server level. However GCC website would give an interface that would more or less follow the Taxonomy.
30
![Page 31: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/31.jpg)
31
Adoption of the proposed product taxonomy.
GCC applying Taxonomy directly on the Catalog.
New organizing of the products on the THREDDS catalog
based on proposed taxonomy
Not necessary and need resources.
NetCDF file content organization of new products
GDWG would look into it and asses its feasibility.
Formats and procedure for storing and exchanging satellite
instrument event logs.
Revisit GSICS Product Acceptance.
Is Big Data an answer to our questions.
![Page 32: GSICS Data Management Working Group](https://reader036.vdocuments.mx/reader036/viewer/2022062315/56815ebc550346895dcd4102/html5/thumbnails/32.jpg)
5-8 March 2012 GSICS Joint Working Groups Meeting, Beijing, China 32