the international tigge archive
DESCRIPTION
The International TIGGE Archive. Baudouin Raoult, Manuel Fuentes - ECMWF Steven Worley, Doug Schuster - NCAR Xiaofeng Bian, Li Xiang - CMA Many Others!. Background: the TIGGE database. Global ensemble forecasts to around 14 days generated routinely at different centres around the world - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/1.jpg)
3rd Thorpex Int. Science Symp., May 2009
The International TIGGE Archive
Baudouin Raoult, Manuel Fuentes - ECMWFSteven Worley, Doug Schuster - NCAR
Xiaofeng Bian, Li Xiang - CMAMany Others!
![Page 2: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/2.jpg)
Background: the TIGGE database
• Global ensemble forecasts to around 14 days generated routinely at different centres around the world
• Three archive centres: CMA, NCAR and ECMWF• Ten data providers:
– ECMWF, JMA (Japan), UK Met Office (UK), CMA (China), NCEP (USA), MSC (Canada), Météo-France (France), BOM (Australia), KMA (Korea), CPTEC (Brazil)
• Data exchanges using UNIDATA LDM, HTTP and FTP• 200 TB, growing by > 1 TB/week
![Page 3: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/3.jpg)
3rd Thorpex Int. Science Symp., May 2009
Background, Brief Time Line– March 2005, TIGGE Workshop at ECMWF
• Collect community input for TIGGE science and data requirements• Choose centralized archives for early start (ECMWF, NCAR, CMA)
– September 2005, meeting of technical experts at ECMWF• Representatives from ECMWF, NCAR, CMA, and NAEFS• Create technical implementation plan to address TIGGE needs
– November 2005, TIGGE Working Group at ECMWF• Present and get approval for the technical implementation plan
– October 2006, first EPS data is delivered– May 2006, User access to TIGGE archive starts– February 2008, all 10 international providers sending EPS
dataSignificant accomplishment, create 24x7 international
system in less than three years
![Page 4: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/4.jpg)
Quality assurance: homogeneity
• Homogeneity is paramount for TIGGE to succeed– The more consistent the archive the easier it will be to
develop applications• There are three aspects to homogeneity:
– Common terminology (parameter names, file names,…)– Common data format (format, units, …)– Definition of an agreed list of products (Parameters, Steps,
levels, …)• What is not homogeneous:
– Resolution– Base time (although most provider have a run at 12 UTC)– Forecast length– Number of ensemble
![Page 5: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/5.jpg)
Strong governance
• Precise definition of:– Which products: list of parameters, levels, steps, units,…– Which format: GRIB2– Which transport protocol: UNIDATA’s LDM– Which naming convention: WMO file name convention
• Only exception: the grid and resolution– Choice of the data provider. Data provider to provide interpolation to
regular lat/lon– Best possible model output
• Many tools and examples:– Sample dataset available– Various GRIB2 tools, “tigge_check” validator, …– Scripts that implement exchange protocol
• Web site with documentation, sample data set, tools, news….
![Page 6: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/6.jpg)
QA: Checking for homogeneity
• E.g. Cloud-cover: instantaneous or six hourly?
![Page 7: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/7.jpg)
Completeness
• The objective is to have 100% complete datasets at the Archive Centres
• Completeness may not be achieved for two reasons: – The transfer of the data to the Archive Centre fails– Operational activities at a data provider are interrupted
and back filling past runs is impractical• Incomplete datasets are often very difficult to use• Most of the current tools used for ensemble
forecasts assume a fixed number of members from day to day
![Page 8: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/8.jpg)
QA: Checking completeness
![Page 9: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/9.jpg)
Archive Status and Monitoring, Variability between providers
![Page 10: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/10.jpg)
Archive Status and Monitoring, Archive Completeness
PL = Pressure Level, PT = 320K θ Level, PV = ± 2 Potential Vorticity Level, SL = Single/Surface Level
Variable LvL ECWF UKMO JMA NCEP CMA CMC BOM MetF KMA CPTC
Geopotential Z PL
Specific H PL
T PL
U-velocity PL
V-velocity PL
Potential Vor PT
Potential T PV
U-velocity PV
V-Velocity PV
U 10m SL
V 10m SL
CAPE SL
Conv. Inhib. SL
Land-sea SL
Mean SLP SL
Orog. SL
Skin T SL
Snow D. H20 SL
Snow F. H20 SL
![Page 11: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/11.jpg)
Archive Status and Monitoring, Archive Completeness
Variable LvL ECWF UKMO JMA NCEP CMA CMC BOM MetF KMA CPTCSoil Moist. SL
Soil T SL
Sunshine D. SL
Surf. DPT SL
Surf. ATmax SL
Surf. ATmin SL
Surf. AT SL
Surf. P SL
LW Rad. Out SL
LH flux SL
Net Rad SL
Net Therm. Rad SL
Sensible Rad. SL
Cloud Cov SL
Column Water SL
Precipitation SL
Wilt. Point SL
Field Cap. SL
PL = Pressure Level, PT = 320K θ Level, PV = ± 2 Potential Vorticity Level, SL = Single/Surface Level
![Page 12: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/12.jpg)
Access to TIGGE data
• Data is available to Research and Education after 48 hours– Self registration by agreeing to the terms and conditions
• Portals at CMA, NCAR and ECMWF
![Page 13: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/13.jpg)
User Data Access and Support @ ECMWF
• TIGGE web site– Documentation– Model changes
• ECMWF portal offers:– Access to offline data– Aggregation along any axis (date, level, parameter, origin, ensemble,
…) – Provision of multi-model data on a single grid (regridding to any
lat/lon grid)– Sub-area selection– Reduces volumes to be downloaded by many order of magnitude
![Page 14: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/14.jpg)
TIGGE @ ECMWF
![Page 15: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/15.jpg)
TIGGE portal at ECMWF (using Ajax)
![Page 16: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/16.jpg)
TIGGE Portal: grid selection
![Page 17: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/17.jpg)
TIGGE Portal: area selection
![Page 18: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/18.jpg)
TIGGE @ ECMWF
![Page 19: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/19.jpg)
TIGGE @ ECMWF: Scheduling users’ requests
• Monitoring• Per user limits• Global limits
![Page 20: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/20.jpg)
TIGGE @ ECMWF Batch access
![Page 21: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/21.jpg)
TIGGE@ECMWF: direct access to offline data (MARS)
![Page 22: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/22.jpg)
3rd Thorpex Int. Science Symp., May 2009
User Data Access and Support, differences @ NCAR
• User data selection and subsetting equivalent to ECMWF
• 2.5 Weeks of Data Online• Archive organized in file groups by provider and
forecast initialization time– Pressure Level, Θ Level, PV Level, Single/Surface Level
• Multiple file download option – Direct through browser, using prepared ‘wget’ script
• Long-term file groups available by request
![Page 23: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/23.jpg)
3rd Thorpex Int. Science Symp., May 2009
User Data Access and Support, differences @ NCAR
• Subset data format in GRIB2 or netCDF• Online queue to monitor subset data processing
progress• Subsetting completion email notification for users• TIGGE Research Support
– Library listing of software tools for data access and analysis– TIGGE Users Group web forum
• Community web space to share questions and findings• Provided by Unidata
• Archive of Tropical Cyclone (cxml) data for participating providers– Updated 6-hourly
![Page 24: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/24.jpg)
3rd Thorpex Int. Science Symp., May 2009
User Data Access and Support, differences @ CMA
• Collaborated with ECMWF and CNIC
• Grid Middleware– Accessing the distributed
TIGGE data based on Grid middleware (GRIA, GOS) technologies
• Production Generation– Selecting the product and
associated parameters based on the product catalogue
– Available product in ensemble mean
• Request Monitoring and Query
• Access Control
![Page 25: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/25.jpg)
User Data Access and Support, differences @ CMA
• Data Retreival– Data Selection
• Selecting different data attributes, such as origin centre, date, base time, forecast time step, forecast type, level type, level and parameter.
• The different data options will be updated dynamically based on the user selection and on the constraints described in the data catalogue.
– Submit data request• The total number of data fields to be retrieved.
– If the number of requested fields is larger than the allowed limit, the portal will suggest the users to split their data retrieval.
– Custom result products• area selection• grid conversion• Plotting
– Browse and download result products
![Page 26: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/26.jpg)
User Data Access and Support, differences @ CMA
• Production Generation– product selection
• Selecting the product and associated parameters based on the product catalogue• Submit product generation request
– Available product in ensemble mean
• Request Monitoring and Query– Check the real-time updates of the information about requests– Access the detailed monitoring information of requests for a period of time,
including request creation time, request finish time, process status, request condition, detailed processing information and result download.
– Query monitoring information according to the request date, request user id, request type and result status.
• Access Control– Four kinds of members– Different member has different role
![Page 27: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/27.jpg)
User Metrics at ECMWF
Active users
0
5
10
15
20
25
30
35
40
45
50
Dec-06
Jan-07
Feb-07
Mar-07
Apr-07
May-07
Jun-07
Jul-07
Aug-07
Sep-07
Oct-07
Nov-07
Dec-07
Jan-08
Feb-08
Mar-08
Apr-08
May-08
Jun-08
Jul-08
Aug-08
Sep-08
![Page 28: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/28.jpg)
User Metrics at ECMWF : User Metrics at ECMWF
![Page 29: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/29.jpg)
User Metrics at ECMWF : Data volumes
Data volumes (GBytes)
0
1000
2000
3000
4000
5000
6000
Dec-06
Jan-07
Feb-07
Mar-07
Apr-07
May-07
Jun-07
Jul-07
Aug-07
Sep-07
Oct-07
Nov-07
Dec-07
Jan-08
Feb-08
Mar-08
Apr-08
May-08
Jun-08
Jul-08
Aug-08
Sep-08
RetrievedDelivered
![Page 30: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/30.jpg)
3rd Thorpex Int. Science Symp., May 2009
Introduction to Poster Session demonstrationsInternational TIGGE Archive Centers
ECMWF http://tigge.ecmwf.int
NCAR http://tigge.ucar.edu
CMA http://wisportal.cma.gov.cn/tigge
Poster Sessions Demonstrations• When
• Tuesday, 15:00-16:30 (immediately following presentation)• Wednesday, 16:30-18:00• Any time by appointment, please ask representatives for assistance
(Raoult, Worley, Bian, Schuster, Fuentes, other person from CMA?)• Purposes
• Illustrate all features of the portals• Register users• Submit data requests• Demonstrate analysis tools
![Page 31: The International TIGGE Archive](https://reader036.vdocuments.mx/reader036/viewer/2022081507/5681623e550346895dd2751e/html5/thumbnails/31.jpg)
Credits
• Alex de Almeida Fernandes, Baudouin Raoult, Beth Ebert, Bian Xiaofeng, Dan Swank, Dave Stepaniak, David Ian Brown, Don Middleton, Dong-Joon Kim, Doug Schuster, Glenn Rutledge, Hannah Wilcox, Jean Nicolau, Jim Fraser, Joerg Urban, Julio Pablo Reyes Fernandez, Justin Cooke, Lang Hongliang, Lewis Poulin, Li Xiang, Ma Qiang, Manuel Fuentes, Michel Grenier, Okki Lee, Richard Dare, Richard Swinbank, Shintaro Yokoi, Simon Thompson, Steve Emmerson, Steve Worley, Steven Anthony, Tan Le, Tian Hao, Tom Yoksas, Waldenio Almeida, Yang Xin, Young-Youn Park, Yves Pelletier, …
• ... and many more