![Page 1: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/1.jpg)
Slide 1 Verification Workshop –Berlin – 11 May 2017
Sub-seasonal to seasonal forecast Verification
Frédéric Vitart and Laura Ferranti
European Centre for Medium-Range Weather Forecasts
![Page 2: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/2.jpg)
Slide 2 Verification Workshop –Berlin – 11 May 2017
INDEX
1. Context: S2S prediction
2. Issues with S2S verification
Space/Time Averaging
Conditional skill
Use of re-forecasts for calibration and verification
3. Verification of weather regime transitions
4. Extreme weather verification
![Page 3: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/3.jpg)
Slide 3 Verification Workshop –Berlin – 11 May 2017
S2S Prediction
![Page 4: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/4.jpg)
Slide 4 Verification Workshop –Berlin – 11 May 2017
A particularly difficult time range: Is it an atmospheric initial condition problem as medium-range forecasting or is it a boundary condition problem as seasonal forecasting? “Predictability Desert”
Some sources of predictability :
Madden Julian Oscillation
ENSO
Land surface conditions: snow-soil moisture
Stratospheric variability
Atmospheric dynamical processes (Rossby wave propagations, weather regimes…)
Sea ice cover –thickness ?
Bridging the gap between Climate and weather prediction
Skill depends on “windows of opportunity”!
![Page 5: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/5.jpg)
Slide 5 Verification Workshop –Berlin – 11 May 2017
Madden-Julian Oscillation and its impacts
The Madden-Julian Oscillation (MJO) is the
major fluctuation in tropical weather on
weekly to monthly timescales. The MJO can
be characterised as an eastward moving
'pulse' of cloud and rainfall near the equator
that typically recurs every 30 to 60 days.
![Page 6: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/6.jpg)
Slide 6 Verification Workshop –Berlin – 11 May 2017
Madden-Julian Oscillation (D. Waliser and S. Woolnough)
Monsoons (H. Hendon)
Africa (A. Robertson and R. Graham)
Extremes (F. Vitart)
Verification and Products (C. Coelho)
Su
b-P
roje
cts
S2S Database
Teleconnections (C. Stan and H. Lin)
WWRP/WCRP Sub-seasonal to Seasonal (S2S) Prediction Project
Research Issues
• Predictability
• Teleconnection
• O-A Coupling
• Scale interactions
• Physical processes
Modelling Issues
• Initialisation
• Ensemble generation
• Resolution
• O-A Coupling
• Systematic errors
• Multi-model combination
Needs & Applications
Liaison with SERA
(Working Group on
Societal and Economic
Research Applications)
![Page 7: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/7.jpg)
Slide 7 Verification Workshop –Berlin – 11 May 2017
Time-
range
Resol. Ens. Size Freq. Hcsts Hcst length Hcst Freq Hcst Size
ECMWF D 0-46 T639/319L91 51 2/week On the fly Past 20y 2/weekly 11
UKMO D 0-60 N216L85 4 daily On the fly 1993-2015 4/month 3
NCEP D 0-44 N126L64 4 4/daily Fix 1999-2010 4/daily 1
ECCC D 0-32 0.45x0.45 L40 21 weekly On the fly 1995-2014 weekly 4
BoM D 0-60 T47L17 33 2/weekly Fix 1981-2013 6/month 33
JMA D 0-34 T319L60 25 2/weekly Fix 1981-2010 3/month 5
KMA D 0-60 N216L85 4 daily On the fly 1996-2009 4/month 3
CMA D 0-45 T106L40 4 daily Fix 1886-2014 daily 4
CNRM D 0-32 T255L91 51 weekly Fix 1993-2014 2/monthly 15
CNR-ISAC D 0-32 0.75x0.56 L54 40 weekly Fix 1981-2010 6/month 1
HMCR D 0-63 1.1x1.4 L28 20 weekly Fix 1981-2010 weekly 10
WWRP/WCRP S2S Database
s2s.ecmwf.int s2s.cma.cn
![Page 8: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/8.jpg)
Slide 8 Verification Workshop –Berlin – 11 May 2017
Sub-seasonal verification
S2S forecasts are based on ensemble forecasts. Metrics used to verify S2S
forecasts include:
RMSE/correlations (MJO/ENSO…)
Reliability diagrams/BS
RPS
CRPS
ROC area
Potential Economic value
….
Usually applied on weekly means/monthly means
![Page 9: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/9.jpg)
Slide 9 Verification Workshop –Berlin – 11 May 2017
Wheeler and Hendon MJO Index
Combined EOF1
Combined EOF2
From Wheeler and Hendon, BMRC
![Page 10: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/10.jpg)
Slide 10 Verification Workshop –Berlin – 11 May 2017
MJO FORECAST
-4 -3 -2 -1 0 1 2 3 4
RMM1
-4
-3
-2
-1
0
1
2
3
4
RM
M2
FORECAST BASED 15/05/1997 00UTC
ECMWF MONTHLY FORECASTS
and Africa
West Hem.
Continent
Maritime
Pacific
Western
Ocean
Indian
2
1
8
7 6
5
4
3
Day 1 Day 5 Day 10
Day 15 Day 20 Analysis
Ens. Mean Verification
![Page 11: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/11.jpg)
Slide 11 Verification Workshop –Berlin – 11 May 2017
Bivariate Correlation with ERA Interim – Ensemble Mean
1999-2010 re-forecasts
![Page 12: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/12.jpg)
Slide 12 Verification Workshop –Berlin – 11 May 2017
Skill of the ECMWF Extended-range forecasts
ROC area: 2-meter temperature in the upper tercile
Day 19-25 Day 26-32
Day 5-11 Day 12-18
![Page 13: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/13.jpg)
Slide 13 Verification Workshop –Berlin – 11 May 2017
S2S verification
Important challenges with S2S verification:
• Extended-range forecasts have very little skill to predict the day to day
variability of the weather. There is a need to verify S2S forecasts over
longer time period and larger domains. What is the optimum space/time
filtering?
• Forecast skill is very flow dependent. Need for conditional verification on
MJO, ENSO, NAO, IOD and SAM phases as well as on particular
weather regimes
• Models drift quickly towards there own climatology. Calibration is
necessary. Operational centres produce re-forecasts to calibrate real-
time S2S forecasts and also for skill assessment.
![Page 14: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/14.jpg)
Slide 14 Verification Workshop –Berlin – 11 May 2017
Buizza and Leutbecher, 2015
The predictability limit is
the time when the
forecast error crosses a
certain threshold. As
threshold, m ‐ 2 σ was
used, where m is the
average climatological
error.
(Z500, T850, U950, V850) and three regions (NH, SH, TR).
Depends on
variables,
regions, spatial
filtering
![Page 15: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/15.jpg)
Slide 15 Verification Workshop –Berlin – 11 May 2017
Spatial Filtering
![Page 16: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/16.jpg)
Slide 16 Verification Workshop –Berlin – 11 May 2017
Seamless prediction and verification
Wheeler et al, 2016
![Page 17: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/17.jpg)
Slide 17 Verification Workshop –Berlin – 11 May 2017
Example of seamless Verification
Wheeler et al, 2016Maps of CORa actual skill for precipitation
Short range
Medium
range
Extended
range
1d1d
1w1w
4w4w
![Page 18: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/18.jpg)
Slide 18 Verification Workshop –Berlin – 11 May 2017
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
forecast probability
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
ob
s fr
equ
ency
0.04
Impact of MJO on S2S skill scores
Reliability DiagramProbability of 2-m temperature in the upper tercileDay 19-25
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
forecast probability
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
ob
s f
req
uen
cy
Europe
0.03
-0.09
MJO in IC NO MJO in IC
N. Extratropics
-0.06
Vitart and Molteni, 2010
![Page 19: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/19.jpg)
Slide 19 Verification Workshop –Berlin – 11 May 2017From Tripathi et al. (2015)
Impact of SSWs on forecast skill scores
Conditional verification is useful:
Better understanding the
contribution of climate drivers
in the model
For users to have more/less
confidence in a forecast a
priori.
This type of verification needs
adequate samples (including re-
forecasts) to allow sub-setting of
the data to provide meaningful
verification.
![Page 20: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/20.jpg)
Slide 20 Verification Workshop –Berlin – 11 May 2017
Biases (eg 2mT as shown here) can have a magnitude larger than the anomalies we
want to predict
Need to calibrate extended-range forecasts
Model Bias (1996-2015) Forecast anomalies
2m-temp forecast day 26-32
1st August start dates
![Page 21: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/21.jpg)
Slide 21 Verification Workshop –Berlin – 11 May 2017
Time-
range
Resol. Ens. Size Freq. Hcsts Hcst length Hcst Freq Hcst Size
ECMWF D 0-46 T639/319L91 51 2/week On the fly Past 20y 2/weekly 11
UKMO D 0-60 N216L85 4 daily On the fly 1993-2015 4/month 3
NCEP D 0-44 N126L64 4 4/daily Fix 1999-2010 4/daily 1
ECCC D 0-32 0.45x0.45 L40 21 weekly On the fly 1995-2014 weekly 4
BoM D 0-60 T47L17 33 2/weekly Fix 1981-2013 6/month 33
JMA D 0-34 T319L60 25 2/weekly Fix 1981-2010 3/month 5
KMA D 0-60 N216L85 4 daily On the fly 1996-2009 4/month 3
CMA D 0-45 T106L40 4 daily Fix 1986-2014 daily 4
CNRM D 0-32 T255L91 51 weekly Fix 1993-2014 2/monthly 15
CNR-ISAC D 0-32 0.75x0.56 L54 40 weekly Fix 1981-2010 6/month 1
HMCR D 0-63 1.1x1.4 L28 20 weekly Fix 1981-2010 weekly 10
WWRP/WCRP S2S Database
![Page 22: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/22.jpg)
Slide 22 Verification Workshop –Berlin – 11 May 2017
The ECMWF ENS re-forecast suite to estimate the M-climate
20y
51Tco639
L91
51Tco319
L912016
11 1111 11
11 1111 11
11 11
… 28 6 2 5 May 9 12
2015
11 1111 11
11 1111 11
11 11
11 1111 5
11 1111 11
11 11
2014
11 1111 11
11 1111 11
11 11
2013
1996
…..
Initial conditions:
ERA Interim+
ORAS4 ocean Ics+
Soil reanalysis
Perturbations:
SVs+EDA(2016)+SPPT+SKEB
![Page 23: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/23.jpg)
Slide 23 Verification Workshop –Berlin – 11 May 2017
MJO WH index bivariate correlation
ECMWF Real-time forecasts - NDJFM 2002-2016
![Page 24: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/24.jpg)
Slide 24 Verification Workshop –Berlin – 11 May 2017
ECMWF Real-time forecasts - NDJFM 2002-2016
Small sample size (~20 cases)
MJO skill varies from year to year (e.g. impact of ENSO)
MJO WH index bivariate correlation
![Page 25: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/25.jpg)
Slide 25 Verification Workshop –Berlin – 11 May 2017
Re-forecasts - common period 1995-2001
MJO WH index bivariate correlation
![Page 26: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/26.jpg)
Slide 26 Verification Workshop –Berlin – 11 May 2017
CY31r1
CY32r2
CY32r3
CY31R1: Parameterisation of ice supersaturation
CY32R2: McRAD (radiation scheme)
CY32R3: Changes in convective scheme (Bechtold at al. 2008)
CY40R1: Improved diurnal cycle of precipitation
CY41R1: revised organized convective detrainment and the revised convective momentum
transport. …
CY40r1
CY41r1
Tl159 Tl255 Tl255 Tl319
60 91 levels
Coupling day 040 62 levels
Improvements in MJO Prediction mostly due to changes in convective parameterization
15
days
MJO WH index bivariate correlation
![Page 27: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/27.jpg)
Slide 27 Verification Workshop –Berlin – 11 May 2017
Performance of the extended-range Forecasts
Day 12-18 Day 19-25 Day 26-32
2-metre temperature RPSS over Northern Extratropics
Vitart, 2014
![Page 28: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/28.jpg)
Slide 28 Verification Workshop –Berlin – 11 May 2017
Issues with re-forecast verification
Re-forecast initialization (re-analyses) is different from the initialization of real-time
forecasts (operational analyses)
Re-forecast ensemble size is often small (typically 5 members) compared to real-
time forecasts. Skill is likely to be underestimated.
Number of re-forecast years is generally too small for properly sampling events like
ENSO.
![Page 29: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/29.jpg)
Slide 29 Verification Workshop –Berlin – 11 May 2017
Brier Skill Score (BSS) with
climatology as a reference.
Two versions of the BSS are
used:
- the uncorrected BSS as
estimated from the
ensemble directly.
- an analytical correction
of the BSS extrapolating
towards a hypothetical
infinite ensemble size
(Ferro, 2007) .
Impact of re-forecast ensemble Size
Ferranti, Corti,
Weisheimer
![Page 30: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/30.jpg)
Slide 30 Verification Workshop –Berlin – 11 May 2017
Verification of Weather regime transition(L. Ferranti)
![Page 31: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/31.jpg)
Slide 31 Verification Workshop –Berlin – 11 May 2017 October 29, 2014
Regimes based on clustering of daily anomalies for 29 cold seasons (1980-2008)
31
‘k means’ clustering applied
to EOF pre-filtered data
(retaining 80% of variance)
m2s2
500 hPa geopotential
NAO+ NAO-
European blocking Atlantic ridge
![Page 32: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/32.jpg)
Slide 32 Verification Workshop –Berlin – 11 May 2017 October 29, 2014
Predicting skill (CRPSS) associated with the Euro-Atlantic Regimes:
32
NAO +
Blocking Atlantic Ridge
Bom
NAO -Bom
![Page 33: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/33.jpg)
Slide 33 Verification Workshop –Berlin – 11 May 2017 October 29, 2014
Trajectories in phase space (c.f. MJO propagation)
33
• ±EOF1 and +EOF2
represent quite well
±NAO and BL
• Trajectories in phase
space summarise
regime evolution
• Unlike MJO, no
preferred direction
Winter 2009/10 Winter 2013/14
EOF1 EOF2
BL: record-breaking
cold temperatures over
Europe
+NAO: exceptional
storminess, but mild
temperatures over
Europe
Based on 5-day running
means
Blocking
NA
O- N
AO
+
How can we evaluate the model ability in predicting regimes transitions?
![Page 34: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/34.jpg)
Slide 34 Verification Workshop –Berlin – 11 May 2017 October 29, 2014
Regime transitions:
34
![Page 35: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/35.jpg)
Slide 35 Verification Workshop –Berlin – 11 May 2017
Extreme event verification
![Page 36: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/36.jpg)
Slide 36 Verification Workshop –Berlin – 11 May 2017
![Page 37: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/37.jpg)
Slide 37 Verification Workshop –Berlin – 11 May 2017
Extreme Forecast Index (EFI)ensemble predictions for 29 June - 5 July 2015
Climate 15 June 2015 18 June 2015 22 June 2015 25 June 2015
(15-21d) (12-18d) (8-14d) (5-11d)
Observed
anomaly
2m temp Cumulative Distribution Function
![Page 38: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/38.jpg)
Slide 38 Verification Workshop –Berlin – 11 May 2017
Example of heat wave verification
(Excess Heat Factor index > 0)
ROC area of the probability of the occurrence of a heatwave
(Excess Heat Factor index > 0) [From Hudson and Marshall 2016)
For a heatwave to be
present, the temperature
averaged over three
consecutive days has to
be greater than the
climatological 95th
percentile (T95) of daily
mean temperature for a
given region.
![Page 39: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/39.jpg)
Slide 39 Verification Workshop –Berlin – 11 May 2017
S2S prediction/verification of tropical cyclones
Prediction of more/less TC activity over a sufficiently large area
and time window.
Justification: TC genesis is strongly modulated by various
models of variability: ENSO, MJO, IOD…. Which can be
predicted by models weeks in advance.
Leroy and Wheeler 2008
MJO Phase 2-3
MJO Phase 6-7
![Page 40: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/40.jpg)
Slide 40 Verification Workshop –Berlin – 11 May 2017
Verification of statistical TC genesis forecast
Leroy and Wheeler 2008
![Page 41: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/41.jpg)
Slide 41 Verification Workshop –Berlin – 11 May 2017
Tropical Storm sub-seasonal Verification
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Forecast Probability
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Ob
se
rve
dF
req
ue
nc
y
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Forecast Probability
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Ob
serv
ed
Fre
qu
en
cy
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Forecast Probability
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Ob
se
rve
dF
req
uen
cy
a) Day 1-7
b) Day 8-14
c) Day 15-21
ECMWF STAT CECMWF
Verification over the Southern Hemisphere as in Leroy et al (2008):
Probability of a TC occurrence during a weekly period over 20x15 degree domains
Vitart, Leroy and Wheeler, MWR 2010
![Page 42: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/42.jpg)
Slide 42 Verification Workshop –Berlin – 11 May 2017
Conclusions S2S prediction and verification is still in an early stage
Re-forecasts needed for model calibration and skill assessment
A main challenge for S2S verification is computing forecastprobabilities under limited (small) ensemble sizes and sometimesrelatively small number of re-forecast years in the hindcasts.
How can we best address verification in a seamless manner, forcomparing forecasts across timescales?
The S2S database represents an important resource for inter-comparison of S2S forecasting systems and evaluation of thebenefits of the multi-model ensemble approach.
![Page 43: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/43.jpg)
Slide 43 Verification Workshop –Berlin – 11 May 2017
Opportunity to use information on multiple time scales
Red Cross - IRI example
![Page 44: Sub-seasonal to seasonal forecast Verification · Slide 1 Verification Workshop –Berlin –11 May 2017 Sub-seasonal to seasonal forecast Verification Frédéric Vitart and Laura](https://reader030.vdocuments.mx/reader030/viewer/2022041207/5d61905588c993bd748baa0f/html5/thumbnails/44.jpg)
Slide 44 Verification Workshop –Berlin – 11 May 2017
Example of heat wave verification
(Excess Heat Factor index > 0)
ROC area of the probability of the occurrence of a heatwave
(Excess Heat Factor index > 0) [From Hudson and Marshall 2016)