calibration in ecmwf · ppt24 > 0.1 mm ppt24 > 10 mm ppt24 > 20 mm ppt24 > 5 mm...
TRANSCRIPT
![Page 1: Calibration in ECMWF · PPT24 > 0.1 mm PPT24 > 10 mm PPT24 > 20 mm PPT24 > 5 mm Relative economic value - Higher relative economic value in the calibrated forecast than in the raw](https://reader036.vdocuments.mx/reader036/viewer/2022070812/5f0afee67e708231d42e579a/html5/thumbnails/1.jpg)
© ECMWF June 7, 2018
Calibration in ECMWF
24-h precipitation in dual ENS resolution forecast
Estibaliz Gascon
David Lavers, David Richardson, Martin Leutbecher, Zied Ben
Bouallegue, Florian Pappenberguer, Tom Hamill (NOAA)
ECMWF
![Page 2: Calibration in ECMWF · PPT24 > 0.1 mm PPT24 > 10 mm PPT24 > 20 mm PPT24 > 5 mm Relative economic value - Higher relative economic value in the calibrated forecast than in the raw](https://reader036.vdocuments.mx/reader036/viewer/2022070812/5f0afee67e708231d42e579a/html5/thumbnails/2.jpg)
Why we should apply a calibration to the ensemble forecast?
• Raw precipitation forecasts are less useful
than they could be because:
– Imperfections in the prediction system.
– Location-dependent and location-independent
biases in the forecast
– Biases may also differ between light and heavy
precipitation events (i.e. overforecasting light
precipitation and underforecasting the heavier)
• For these reasons, statistical postprocessing is
often applied.
– The method applied here is quantile mapping.
(keep the spatial distribution of the field)
2EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
Figure adapted from Hamill et al. (2017)
![Page 3: Calibration in ECMWF · PPT24 > 0.1 mm PPT24 > 10 mm PPT24 > 20 mm PPT24 > 5 mm Relative economic value - Higher relative economic value in the calibrated forecast than in the raw](https://reader036.vdocuments.mx/reader036/viewer/2022070812/5f0afee67e708231d42e579a/html5/thumbnails/3.jpg)
Quantile mapping applied in ECMWF 24h-h precipitation
3EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
DATABASES ECMWF experiment
Observation/ analysis
database
• EFAS (European Flood Awareness System) 24h precipitation 5
km analysis
• 20 years from 1996 to 2015
Supplemental locations
• 50 supplemental locations for each grid point.
• Based on Hamill et al. (2017).
• Applied to 20 years of EFAS 5km precipitation analysis
Reforecast database for quantile
mapping
• Re-forecast interpolated to 5 km.
• 50 supplemental locations.
• 20 years x 9 runs x 50 sup.loc x 1 cf = 9000 samples
Climatology database for
quantile mapping• EFAS 24h precipitation
• 50 sup.loc x 20 years x 9 runs = 9000 samples
Location-
dependent biases
Location-
independent biases
![Page 4: Calibration in ECMWF · PPT24 > 0.1 mm PPT24 > 10 mm PPT24 > 20 mm PPT24 > 5 mm Relative economic value - Higher relative economic value in the calibrated forecast than in the raw](https://reader036.vdocuments.mx/reader036/viewer/2022070812/5f0afee67e708231d42e579a/html5/thumbnails/4.jpg)
Dual ENS calibration tests
4EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
RAW operational (~18 km) 51 ENS members
RAW low resolution (~28 km) 201 ENS members
RAW DUAL ENS Combinations
• (51,0)• (40,40)• (20,120)• (10,160)• (0,201)
CAL operational (5 km) 51 ENS members
CAL low resolution (5 km) 201 ENS members
CALIBRATED DUAL ENS Combinations
• (51,0)• (40,40)• (20,120)• (10,160)• (0,201)
CALIBRATIONQuantile mapping
- All the ENS combinations have the same computational cost than the current operational ENS system (0,51)
- Calibration applied to each ENS member independently.
![Page 5: Calibration in ECMWF · PPT24 > 0.1 mm PPT24 > 10 mm PPT24 > 20 mm PPT24 > 5 mm Relative economic value - Higher relative economic value in the calibrated forecast than in the raw](https://reader036.vdocuments.mx/reader036/viewer/2022070812/5f0afee67e708231d42e579a/html5/thumbnails/5.jpg)
VERIFICATION
• 24h total precipitation June, July and August
2016 across Europe
• EFAS 24h precipitation at SYNOP locations.
• Lead times day 1 ,3, 5, 7 and 10
• Verify the ENS combinations (0,201),
(10,160), (20,120), (40,40) and (51,0)
EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
STUDY AREA
![Page 6: Calibration in ECMWF · PPT24 > 0.1 mm PPT24 > 10 mm PPT24 > 20 mm PPT24 > 5 mm Relative economic value - Higher relative economic value in the calibrated forecast than in the raw](https://reader036.vdocuments.mx/reader036/viewer/2022070812/5f0afee67e708231d42e579a/html5/thumbnails/6.jpg)
Supplemental locations (based on the method from Hamill et al. (2017)
6EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
• To increase the training sample size for the
quantile mapping.
• Reduce the systematic bias and location-
dependant biases from a specific grid point.
• Based on common terrain and weather
features:
- 24-h Precipitation CDFs
- Terrain heights
- Geography (terrain facet)
![Page 7: Calibration in ECMWF · PPT24 > 0.1 mm PPT24 > 10 mm PPT24 > 20 mm PPT24 > 5 mm Relative economic value - Higher relative economic value in the calibrated forecast than in the raw](https://reader036.vdocuments.mx/reader036/viewer/2022070812/5f0afee67e708231d42e579a/html5/thumbnails/7.jpg)
VERIFICATION
7EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
- Better CRPS for all lead times and all
ENS combinations, most significant in
shorter lead times.
- (0, 51) and (40,40) are the best
combinations, in both, RAW and
CALIBRATED forecasts.
- Quite similar score values for all the
combinations at lead times equal or
longer than 5 days.
CRPS
![Page 8: Calibration in ECMWF · PPT24 > 0.1 mm PPT24 > 10 mm PPT24 > 20 mm PPT24 > 5 mm Relative economic value - Higher relative economic value in the calibrated forecast than in the raw](https://reader036.vdocuments.mx/reader036/viewer/2022070812/5f0afee67e708231d42e579a/html5/thumbnails/8.jpg)
VERIFICATION
8EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
Day 1 PPT24>0.1 mm
Day 1 PPT24>10 mm
Day 10 PPT24>0.1 mm
Day 10 PPT24>10 mm
Reliability
- Reliability improves after the
calibration at least up to day
10 lead time and different
PPT24 thresholds.
- Similar results in the current
operational ENS system
(0,51) than the dual ensemble
combinations (i.e. 40,40)
![Page 9: Calibration in ECMWF · PPT24 > 0.1 mm PPT24 > 10 mm PPT24 > 20 mm PPT24 > 5 mm Relative economic value - Higher relative economic value in the calibrated forecast than in the raw](https://reader036.vdocuments.mx/reader036/viewer/2022070812/5f0afee67e708231d42e579a/html5/thumbnails/9.jpg)
VERIFICATION
9EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
Day 1 PPT24>0.1 mm Day 10 PPT24>0.1 mm
ROC curves
- Forecast skill improves after
the calibration at least up to
day 10 lead time and
different PPT24 thresholds.
- Similar results in the current
operational system (0,51)
than the dual ensemble
combinations (i.e. 40,40).
Day 1 PPT24>10 mm Day 10 PPT24>10 mm
![Page 10: Calibration in ECMWF · PPT24 > 0.1 mm PPT24 > 10 mm PPT24 > 20 mm PPT24 > 5 mm Relative economic value - Higher relative economic value in the calibrated forecast than in the raw](https://reader036.vdocuments.mx/reader036/viewer/2022070812/5f0afee67e708231d42e579a/html5/thumbnails/10.jpg)
VERIFICATION
10EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
PPT24 > 0.1 mm
PPT24 > 20 mmPPT24 > 10 mm
PPT24 > 5 mm
Relative economic value
- Higher relative economic value
in the calibrated forecast than
in the raw forecast, at least up
to 5 mm threshold and for all
the lead times.
- A greater number of users with
different C/L can benefit from
the calibrated forecast,
compared to the raw forecast.
![Page 11: Calibration in ECMWF · PPT24 > 0.1 mm PPT24 > 10 mm PPT24 > 20 mm PPT24 > 5 mm Relative economic value - Higher relative economic value in the calibrated forecast than in the raw](https://reader036.vdocuments.mx/reader036/viewer/2022070812/5f0afee67e708231d42e579a/html5/thumbnails/11.jpg)
11EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
RAW CALIBRATED
EFAS 5 km analysisPPT24 > 0.1 mm
1 day lead time
%%
mm
RAW CALIBRATED
EFAS 5 km analysisPPT24 > 0.1 mm
1 day lead time
%
mm
![Page 12: Calibration in ECMWF · PPT24 > 0.1 mm PPT24 > 10 mm PPT24 > 20 mm PPT24 > 5 mm Relative economic value - Higher relative economic value in the calibrated forecast than in the raw](https://reader036.vdocuments.mx/reader036/viewer/2022070812/5f0afee67e708231d42e579a/html5/thumbnails/12.jpg)
12EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
EFAS 5 km analysis
PPT24 > 10 mm
1 day lead time
CALIBRATEDRAW% %
mm
![Page 13: Calibration in ECMWF · PPT24 > 0.1 mm PPT24 > 10 mm PPT24 > 20 mm PPT24 > 5 mm Relative economic value - Higher relative economic value in the calibrated forecast than in the raw](https://reader036.vdocuments.mx/reader036/viewer/2022070812/5f0afee67e708231d42e579a/html5/thumbnails/13.jpg)
13EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
%
RAW CALIBRATED
EFAS 5 km analysis
PPT24 > 0.1 mm
5 day lead time
%
mm
![Page 14: Calibration in ECMWF · PPT24 > 0.1 mm PPT24 > 10 mm PPT24 > 20 mm PPT24 > 5 mm Relative economic value - Higher relative economic value in the calibrated forecast than in the raw](https://reader036.vdocuments.mx/reader036/viewer/2022070812/5f0afee67e708231d42e579a/html5/thumbnails/14.jpg)
CONCLUSIONS
▪ For all lead times and combinations, the calibrated forecast has better and resolution
▪ This calibration especially improves the forecast of low 24-h precipitation thresholds
▪ CRPS score shows that the most skilful combination is (40,40); however, the scores are similar to operational system.
▪ All the combinations have similar values in terms of reliability, skill or relative economic value.
14EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS