goes-r awg product validation tool development...goes-r awg product validation tool development...
TRANSCRIPT
![Page 1: GOES-R AWG Product Validation Tool Development...GOES-R AWG Product Validation Tool Development Hydrology Application Team Bob Kuligowski (STAR) 2 Outline • Products • Validation](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2f63fdff3fdd6d4a1280c7/html5/thumbnails/1.jpg)
1
GOES-R AWG Product Validation Tool Development
Hydrology Application Team
Bob Kuligowski (STAR)
![Page 2: GOES-R AWG Product Validation Tool Development...GOES-R AWG Product Validation Tool Development Hydrology Application Team Bob Kuligowski (STAR) 2 Outline • Products • Validation](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2f63fdff3fdd6d4a1280c7/html5/thumbnails/2.jpg)
2
Outline
• Products
• Validation Strategies
• Routine Validation Tools
• “Deep-Dive” Validation Tools
• Ideas for the Further Enhancement and Utility of Validation Tools
• Summary
![Page 3: GOES-R AWG Product Validation Tool Development...GOES-R AWG Product Validation Tool Development Hydrology Application Team Bob Kuligowski (STAR) 2 Outline • Products • Validation](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2f63fdff3fdd6d4a1280c7/html5/thumbnails/3.jpg)
Products
• Rainfall Rate / QPE – 4-km resolution – every 15 min – FD equatorward of 60°
and with a LZA<70°
3
![Page 4: GOES-R AWG Product Validation Tool Development...GOES-R AWG Product Validation Tool Development Hydrology Application Team Bob Kuligowski (STAR) 2 Outline • Products • Validation](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2f63fdff3fdd6d4a1280c7/html5/thumbnails/4.jpg)
Validation Strategies: Overview
• Twofold validation strategy: – Determine performance of the
algorithm against spec – Identify systematic biases /
weaknesses in the algorithm • Ground validation datasets:
– TRMM Precipitation Radar (35ºS-35ºN) from NASA
– Nimrod radar composite (western Europe) from the British Atmospheric Data Centre
4 Sample Nimrod 3-h accumulation
Sample TRMM rain rates for a 24-h period
![Page 5: GOES-R AWG Product Validation Tool Development...GOES-R AWG Product Validation Tool Development Hydrology Application Team Bob Kuligowski (STAR) 2 Outline • Products • Validation](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2f63fdff3fdd6d4a1280c7/html5/thumbnails/5.jpg)
Validation Strategies
• Routine validation tools: – Time series of accuracy and precision
• Is the algorithm meeting spec on a consistent basis? • Are there any trends in performance that might need to be
addressed even if the algorithm is still meeting spec at this time?
– Spatial plots of rainfall rates vs. ground validation • Are the rainfall rate fields physically reasonable? • Do the rainfall rate fields compare reasonably well with ground
truth? – Scatterplots vs. ground validation
• Are there any anomalous features in the scatterplots that could indicate errors not revealed by the spatial plots?
5
![Page 6: GOES-R AWG Product Validation Tool Development...GOES-R AWG Product Validation Tool Development Hydrology Application Team Bob Kuligowski (STAR) 2 Outline • Products • Validation](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2f63fdff3fdd6d4a1280c7/html5/thumbnails/6.jpg)
Validation Strategies
• Deep-dive validation tools: – Comparing calibration MW data with ground truth
• How much of the error is due to the calibration data rather than the calibration process?
– Divide data by algorithm class and analyze • Are errors in the algorithm associated with a particular
geographic region or cloud type? – Spatially distributed statistics
• Does the algorithm display any spatial biases (e.g., latitudinal, land vs. ocean) that need to be addressed?
– Analyze the rainfall rate equations for particular cases • Are there particular predictors or calibration equations
associated with errors?
6
![Page 7: GOES-R AWG Product Validation Tool Development...GOES-R AWG Product Validation Tool Development Hydrology Application Team Bob Kuligowski (STAR) 2 Outline • Products • Validation](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2f63fdff3fdd6d4a1280c7/html5/thumbnails/7.jpg)
Routine Validation Tools
• Capabilities: – Match Rainfall Rate with ground data (TRMM and
Nimrod radar pre-launch; GPM and Stage IV / MPE post-launch)
– Compute accuracy, precision – Compute basic validation statistics (volume bias,
correlation, and threshold-dependent POD, FAR, area bias, and HSS)
– Create joint distribution files
7
![Page 8: GOES-R AWG Product Validation Tool Development...GOES-R AWG Product Validation Tool Development Hydrology Application Team Bob Kuligowski (STAR) 2 Outline • Products • Validation](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2f63fdff3fdd6d4a1280c7/html5/thumbnails/8.jpg)
Routine Validation Tools
• Use GrADS for all visualization: – Spatial plots of Rainfall Rate and
ground-truth data – Plots of POD, FAR, area bias, and
HSS vs. threshold – Rainfall Rate / ground-truth data
joint distribution
8
![Page 9: GOES-R AWG Product Validation Tool Development...GOES-R AWG Product Validation Tool Development Hydrology Application Team Bob Kuligowski (STAR) 2 Outline • Products • Validation](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2f63fdff3fdd6d4a1280c7/html5/thumbnails/9.jpg)
• Capabilities: – Match Rainfall Rate with ground data (TRMM and
Nimrod radar pre-launch; GPM and Stage IV / MPE post-launch)
– Match calibration MW rainfall rates with ground data – Divide matched Rainfall Rate and ground data by
algorithm class – Divide matched Rainfall Rate and ground data by
location – Compute performance statistics by algorithm class
and location – Extract rain/no rain and rate equations and
distribution adjustment LUTs 9
“Deep-Dive” Validation Tools
![Page 10: GOES-R AWG Product Validation Tool Development...GOES-R AWG Product Validation Tool Development Hydrology Application Team Bob Kuligowski (STAR) 2 Outline • Products • Validation](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2f63fdff3fdd6d4a1280c7/html5/thumbnails/10.jpg)
• Use GrADS for all visualization: – Spatial plots and joint distribution
plots of both calibration MW rain rates and of GOES-R Rainfall Rates vs. ground data
– Spatial plots of performance statistics
10
“Deep-Dive” Validation Tools
![Page 11: GOES-R AWG Product Validation Tool Development...GOES-R AWG Product Validation Tool Development Hydrology Application Team Bob Kuligowski (STAR) 2 Outline • Products • Validation](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2f63fdff3fdd6d4a1280c7/html5/thumbnails/11.jpg)
• Use GrADS for all visualization: – 2-D plots of rain/no rain and rate
values as a function of predictor values – Spatial plots of the training data
(predictors & targets)
11
“Deep-Dive” Validation Tools
![Page 12: GOES-R AWG Product Validation Tool Development...GOES-R AWG Product Validation Tool Development Hydrology Application Team Bob Kuligowski (STAR) 2 Outline • Products • Validation](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2f63fdff3fdd6d4a1280c7/html5/thumbnails/12.jpg)
• A handy tool would be a GUI that would allow the user to select a portion of a Rainfall Rate field and automatically create regional plots of: – Performance statistics vs. ground validation and
available calibration data – Joint distribution – Predictor fields – Rain / no rain and rain rate equations and distribution
adjustment LUTs – etc.
12
Ideas for the Further Enhancement
and Utility of Validation Tools
![Page 13: GOES-R AWG Product Validation Tool Development...GOES-R AWG Product Validation Tool Development Hydrology Application Team Bob Kuligowski (STAR) 2 Outline • Products • Validation](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2f63fdff3fdd6d4a1280c7/html5/thumbnails/13.jpg)
13
Summary
• The GOES-R Rainfall Rate algorithm will be validated against TRMM and Nimrod radar pre-launch and GPM and Stage IV / MPE post-launch
• Validation will focus on evaluating performance and identifying areas for potential improvement – Routine validation will focus on the former using
time series of statistics, spatial plots and joint distribution plots
– Deep-dive validation will focus on the latter by examining the predictor and target data along with the calibration to determine the reasons for any anomalies in performance