A Comparison of Two Postprocessing Approaches as Part of the ‘HEavy Precipitation forecast

Postprocessing over India’ (HEPPI) ProjectMichael Angus, Martin Widmann, Andrew Orr, G.C. Leckebusch

Daily rainfall averaged over July 5th to 8th 2019

Forecast - Observations Corrected Forecast - Observations• Accurate predictions of heavy precipitation in India are vital for impact-orientated forecasting

• Operational forecasts from non-convection-permitting models can have large biases in the intensities and spatial structure of heavy precipitation

• Statistical postprocessing can reduce these biases for relatively little computational cost, but few studies have focused on postprocessing tropical or monsoonal rainfall

vEGU21, AS1.10

Background

Objectives

• Develop several postprocessing methods for precipitation forecast over India

• Evaluate the methods systematically using multiple skill metrics focusing on heavy precipitation

• make the best method ready to be used in operational forecasting at NCMRWF

Schefzik et al. 2017

Bihar Sept. 2019 Uttarakhand June 2013 Kerala August 2018

Datasets

• Forecast: NEPS (NCMRWF): 12 km, 23 ensemble members, May - October 2018, 2019, lead time up to 12 days

• Observations:NCMRWF-IMD merged satellite and rain gauge data at 0.25 degreesIMERG, Integrated Multi-satellitE Retrievals for GPM at12km

Methodology

• Univariate Quantile Mapping (UQM)• Ensemble Member Output Statistics (EMOS)

Experiment Setup

Test Locations

• Postprocessing performed on each 0.25°x.0.25° grid cell in the IMD observed dataset

• To compare marginal distributions, and for certain validation metrics, we select 9 test locations to represent a range of climatologies across India

1. Mumbai, 18.3°N, 73.3°E

2. Rajasthan 25.0°N, 74.0°E

3. Kerala 10.0°N, 76.6°E

4. Shimla 32.4°N, 76.4°E

5. Delhi 28.4°N, 77.1°E

6. Hyderabad 17.5°N, 77.8°E

7. Patna 26.4°N, 84.8°E

8. Bhubaneswar 20.5°N, 85.7°E

9. Meghalaya 25.6°N, 91.5°E.

For reference, the 90th percentile rainfall at each grid cell, as determined from the IMD observations.

Method 1 - Univariate Quantile Mapping (UQM)

Quantile-Quantile plot for raw or postprocessed forecast (specified in legend) and observed rainfall at the specified location. Quantiles are at an interval of 1 from the 1st to 90th percentile, and at 0.1 thereafter.

Standard approach that maps the simulated (NEPS forecast) probability density function (PDF) onto the observed (IMD) PDF

Fitted both Gamma and mixed

distribution, where values of heavy

precipitation (over the local 90th

percentile) are fitted separately

(Pastén-Zapata et al., 2020) - Double

Gamma Quantile Mapping (DGQM).

DGQM consistently offers best

performance – in subsequent

slides, UQM refers to this fitting

Method 2 - Ensemble Member Output Statistics (EMOS)

Spread optimised and adjusted to observation (black line)E

nse

mble

M

em

bers

(n)

Cum

ula

tive D

ensi

ty

Spread across ensemble members at Kerala, June 9th 2018

UQM EMOS

OptimisesEnsemble spread

OptimisesEnsemble member timeseries

EMOS considers the PDFs of the forecast ensemble members at a

given time and determines a transformation of these PDFs such

that the postprocessed PDFs optimally fit the observations.

The method is applied locally, and the transformed ensemble

mean and variance are linear functions of the raw ensemble

mean and variance, using the same regression coefficients for

every timestep (e.g., Gneiting et al. 2005, Wilks 2006, Schefzik

2017).

Example EMOS application at a single location and timestep

Comparison between EMOS and UQM – Distribution vs Spread

EMOS CDF shifted too wet from forecast

UQM Ensemble spread increases where observations are positively biased and decreases where observations are negatively biased

EMOS optimised for ensemble spread

Marginal Distributions

Errors in the PDFs of the raw forecast are consistently reduced by UQM, regardless of whether the forecast is skewed towards too many relatively dry (>0 to 20mm/day) or too many heavy precipitation days

As EMOS optimises ensemble spread it does not guarantee that the postproccessed local distributions are close to the observed ones.

This can be seen at some locations (e.g., Kerala, Hyderabad) where the EMOS PDFs are similar to the raw forecast PDFs and substantially different from the observed PDFs.

Rank Histograms

The EMOS approach corrects for underdispersion, with a mostly even distribution of the observation rank at all test locations.

The UQM approach does not address the underdispersion, and in several cases exacerbates the problem.

For example the forecast is too dry in Delhi for all quantiles, and the UQM postprocessing increases the rainfall throughout.

The EMOS postprocessing is therefore the best choice for correcting under- or overdispersion in forecasts.

For each test location, the rank of the observations with respect to the 23 ensemble members of

the raw and postprocessed forecast is shown as a frequency histogram. Only wet days (more

than 0 mm/day) are included in the relative frequencies). A rank of 1 (24) indicates the observed

rainfall was wetter (drier) than all ensemble members.

Standard Validation metrics – Continuous Ranked Probability Score

Mean difference between observed rainfall and the

probabilistic forecast of all timesteps where rain occurred

in the IMD observations (>0 mm/day). The CDF of the

forecast is compared to a Heaviside function that equals 1

when an event occurs:

𝐶𝑅𝑃𝑆 =1

𝑛

𝑖=1

𝑛

න−∞

+∞

{𝐹𝑖 𝑦 − 𝐼{𝑦≥𝑥𝑖}}2𝑑𝑦

where Fi(y) is the CDF of the forecast rainfall at timestep i, xi is the observed value at timestep i, n is the number of events and I{y≥xi} is the Heaviside function

a) CRPS skill score for the UQM postprocessing. Green values indicate improvedperformance with reference to the raw forecast. b) as in a), but for the EMOSpostprocessing. c) Each grid cell is coloured with reference to the best performingforecast (minimum CRPS): green (1) for the raw forecast, pink (2) for the UQMapproach and cyan (3) for the EMOS approach. d) For reference, the 90th percentilerainfall at each grid cell, as determined from the IMD observations.

EMOS best performing method (bottom left).

UQM only outperforms the raw forecast in areas of high rainfall

Heavy Rainfall Validation metrics – Receiver Operating Characteristic (ROC)

Measures the ability of the forecast to discriminate

between events and non-events, by comparing hit rate to

false alarm rate

In good discrimination forecasts, the ROC curve

approaches the top-left corner (high hit rate, low false

alarm rate for most thresholds). In poor discrimination

forecasts, these curves get close to the line of equality

(no resolution between hits and misses).

Both the raw and postprocessed forecasts show skill

across all test locations. Collectively, the forecasts

perform best in the regions of high rainfall, e.g., Mumbai,

Kerala, and Patna.

In these locations the forecast and both postprocessing

techniques perform similarly, although the UQM curve

is consistently the highest, indicating best

performance.

Receiver Operating Characteristic diagram for heavy rainfall (>90th percentile) at each location. Curves determined by multiple probability thresholds within the ensemble.

Heavy Rainfall Validation metrics – ROC Skill Score

For the whole of India, we calculate the ROC skill

score (RSS) at each grid cell:

𝑅𝑆𝑆 =𝐴−𝐴𝑓

1−𝐴𝑓

where A is the area under the curve of the postprocessed forecast, and Af is the area under the curve of the raw forecast.

a) ROC skill score for the UQM postprocessing. Green values indicate improvedperformance with reference to the raw forecast. b) as in a), but for the EMOSpostprocessing. c) Each grid cell is coloured with reference to the best performingforecast (maximum ROC area): green (1) for the raw forecast, pink (2) for the UQMapproach and cyan (3) for the EMOS approach. d) For reference, the 90th percentilerainfall at each grid cell, as determined from the IMD observations.

The skill score is consistently above zero for both

UQM and EMOS.

UQM performs best throughout India as a

whole.

The raw forecast rarely performs better than both

UQM and EMOS, with the lowest ROC score in the

majority of grid cells.

Heavy Rainfall Validation metrics – Brier Skill Score

The Brier Score (BS) is calculated as:

𝐵𝑆 =1

𝑛σ𝑖=1𝑛 (𝑦𝑖 − 𝑜𝑖)

2

where n is the number of events, yi is the forecast probability and oi is the binary of the ith event (0 for non-occurrence, 1 for occurrence).

Brier skill score for the UQM postprocessing. Green values indicate improvedperformance with reference to the raw forecast. b) as in a), but for the EMOSpostprocessing. c) Each grid cell is coloured with reference to the best performingforecast (minimum brier score): green (1) for the raw forecast, pink (2) for the UQMapproach and cyan (3) for the EMOS approach. d) For reference, the 90th percentilerainfall at each grid cell, as determined from the IMD observations.

The skill score is consistently above zero forEMOS, which performs best throughout Indiaas a whole.

UQM is consistently outperformed by EMOS, and bythe forecast in most locations

Conclusions and Future Outlook

Conclusions

Overall EMOS performs best with respect to correcting the model spread and associated underdispersion, as well as with respect to probabilistic skill scores for all rainfall values (CRPS) and for heavy rainfall (BSS).

UQM performs best with respect to correcting the model PDF in most locations.

The ROC and RSS results are inconclusive and location dependent, although both postprocessing methods consistently outperform the raw forecast. These findings are independent of which of the two gridded rainfall data sets are used for validation (see following slides for alternative IMERG observation validations).

Recommendation

We recommend EMOS for operational implementation, as from a user perspective a good performance in forecasting a given observations (CRPS) and in particular heavy precipitation events (BSS) can be expected to be more important than achieving a close match between the forecasted and observed local precipitation distributions.

Implementation

Selected best scoring method (EMOS) to be implemented using the Met Office IMPROVER Post-processing system

Thank you! Any Questions?

Contact me at M.Angus@bham.ac.uk

References

Baran, S., & Nemoda, D. (2016). Censored and shifted gamma distribution based EMOS model for probabilistic quantitative precipitation forecasting. Environmetrics, 27(5), 280-292.Gneiting, T., Raftery, A. E., Westveld III, A. H., & Goldman, T. (2005). Calibrated probabilistic forecasting using ensemble model output statistics and minimum CRPS estimation. Monthly Weather Review, 133(5), 1098-1118.Javanshiri Z, Fathi M, Mohammadi SA. (2021) Comparison of the BMA and EMOS statistical methods for probabilistic quantitative precipitation forecasting. Meteorol Appl. 28:e1974. https://doi.org/10.1002/met.1974Maraun, D., & Widmann, M. (2018). Statistical downscaling and bias correction for climate research. Cambridge University Press.Pastén-Zapata, E., Jones, J. M., Moggridge, H., & Widmann, M. (2020). Evaluation of the performance of Euro-CORDEX Regional Climate Models for assessing hydrological climate change impacts in Great Britain: A comparison of different spatial resolutions and quantile mapping bias correction methods. Journal of Hydrology, 584, 124653.Schefzik, R. (2017). Ensemble calibration with preserved correlations: unifying and comparing ensemble copula coupling and member‐by‐member postprocessing. Quarterly Journal of the Royal Meteorological Society, 143(703), 999-1008.Wilks, D.S. (2006) Comparison of ensemble MOS methods in the Lorenz 96 setting. Meteorological Applications, 13, 243–256.Wilks, D.S. (2011) Statistical Methods in the Atmospheric Sciences, 3rd edition. New York: Academic Press

Uncertainty estimates – Metrics with IMERG observations

Uncertainty estimates – Metrics with IMERG observations

Mean difference between observed rainfall and the

probabilistic forecast of all timesteps where rain occurred

in the IMD observations (>0 mm/day). The CDF of the

forecast is compared to a Heaviside function that equals 1

when an event occurs:

𝐶𝑅𝑃𝑆 =1

𝑛

𝑖=1

𝑛

න−∞

+∞

{𝐹𝑖 𝑦 − 𝐼{𝑦≥𝑥𝑖}}2𝑑𝑦

where Fi(y) is the CDF of the forecast rainfall at timestep i, xi is the observed value at timestep i, n is the number of events and I{y≥xi} is the Heaviside function

Uncertainty estimates – Metrics with IMERG observations

Receiver Operating Characteristic (ROC)

Uncertainty estimates – Metrics with IMERG observations

Receiver Operating Characteristic Skill

Score (ROCSS)

Uncertainty estimates – Metrics with IMERG observations

The Brier Score (BS) is calculated as:

𝐵𝑆 =1

𝑛σ𝑖=1𝑛 (𝑦𝑖 − 𝑜𝑖)

2

where n is the number of events, yi is the forecast probability and oi is the binary of the ith event (0 for non-occurrence, 1 for occurrence).