Update to COPC:

Global Model Performance Dropouts

Dr. Jordan AlpertNOAA Environmental Modeling Center

contributions from Dr. Brad Ballish, Dr. Da Na Carlis, Dr. Rolf Langland and CDR Mark Moran

2

Outline

• Tools to compare ECMWF and NCEP Dropouts

• Model Runs for Dropout Experiments

• Results from SH Dropout Experiments

• Model Runs for Dropout Experiments

• Comparison of How Well the GSI and ECM “Fit” Obs

• Wind Speed Biases

• How GSI and ECM Fit Observations

3

Poor Forecasts or Skill Score “Dropouts” Lower GFS Performance.

Detailed on next slide

4

Skill Score “Dropouts” LowerOverall GFS Performance.

5

Analysis Team for Dropout Issue

Jordan Alpert - EMCDaNa Carlis - EMCBrad Ballish - NCOKrishna Kumar - NCORolf Langland - NRL

6

Tools to compare ECMWF and NCEP Dropouts

• Use ECMWF analysis to generate “Pseudo Obs” for input to the Gridded Statistical Interpolation (GSI) and GFS forecasts.

• Generate a Climatology of NH and SH dropouts – what are the systematic differences

• Interpolate ECM and GSI analyses to observations to determine comparative strengths and weaknesses.– Statistically analyze observation type fits– stratify by pressure, type, and difference magnitude

Goal: Diagnose Quality Control problems to implement real-time QC detection/correction and improvements to analysis system algorithms

7

Model Runs for Dropout Experiments

• GFS: Operational (Production) GFS

• ECMWF: Operational

• ECM: Pseudo Obs from ECMWF analysis using the GSI as a “Grand Interpolator”

• ecmanlGES: GSI run using ECM as background guess plus GDAS observations

• CNTRL: Updated GSI system with GDAS obs

• InterpGES: Updated GSI with previous 3, 6, & 9-hour ECM forecasts as background guess plus GDAS observations

8

Results from SH Dropout Experiments

• Using ECM (Pseudo Obs) yielded a 90% success rate for alleviating SH dropouts.

• Adding GDAS Obs (ecmanlGES) lowers the success rate to ~75%.• Combining updated GSI system / GDAS obs (CNTRL) with all obs

improves GSI but not enough to alleviate dropouts• Combining background guess from ECM plus all obs (InterpGES )

also improves GSI but not enough to alleviate dropouts• Composites show low-level temperature difference near 850mb in

SH.

Adding the SH observations and the 3, 6, 9-h background guess degrades the ECM forecast

9

Rolf Langland (NRL Monterey)shows systemic height differences between all models and ECMWF (shown is ECMWF-NCEP).

Cause may be the difference in satellite window coverage (under study):

ECMWF (12-h) vs. others (6-h).

Plots at left show height difference plots of time (October to December 2007) vs. longitude, averaged over 35-65N latitudes.

The range of the bias is ±12 m

10

Comparison of How Well the GSI and ECM “Fit” Observations

• Statistics are made on how well the GSI and ECM analyses “fit” the observations for different observation types, as function of pressure, for different regions

• This study shows that each analysis “fits” certain observation types with differing amounts of success, but does not conclude which performs better.

11

Comments on Wind Speed Biases

• The ECM analysis winds at satellite wind locations are stronger than those of the GSI

• The ECMWF data quality control is more aggressive at deleting satellite winds with speeds slower than the background

• ECM wind speeds are stronger as weaker satellite winds are deleted or given less weight in the ECMWF analyses

• For ACARS and sondes, the biases are similar

Work continues to analyze satellite wind speed biases to determine an implementable algorithm for QC, bias correction, and analysis weights

12

How GSI and ECM Fit Observations

• The GSI is found to draw more closely for most winds, especially satellite winds and when there are moderate differences in the analyses (outliers)

• The ECM analyses draw more closely for radiosonde temperatures, especially away from the middle atmosphere and for moderate differences in the analyses

• Fits for surface pressure and moisture are similar – not shown here

• ECM winds are stronger at satellite wind locations based on speed bias stats but with similar fits for radiosonde and ACARS speeds

Changes in the QC and Ob errors could retune these data fits

13

Summary and Future Work

• ECM analysis show dropouts can be alleviated in GFS forecasts

• Running the operational GSI with an ECM derived background guess results in better forecast skill than the operational GFS but not as good as ECM runs

• Running the operational GSI after removing select observation types offers a systematic approach for assessing the impact of different observation types

• There are systematic height differences between all models and ECMWF perhaps from the larger satellite window coverage of ECMWF (12-h) vs others (6-h)

• Work continues to analyze what is the optimal fit of the analysis to observation types and to determine an implementable algorithm for improved QC, bias correction, and analysis weights

• Work has started to use baroclinic instability rates to help analyze dropout cases and analysis differences

14

Background Slides

15

INPUT-----------OUTPUT( )

ECM(WF) Analysis1x1 deg, 15 levels-------------------------

PSEUDO ECM “OBS”

PSEUDO ECM “OBS”GFS GUESS--------------

ECM ANLYSIS“PRE-COND” GUESS

Run GSI

Run GSIagain

ECMWF INITIAL CONDITIONS FOR GFS FORECASTS

“ECM” Runs”

PRE-CONDECM GUESS

PSEUDO ECM “OBS”

---------------ECM ANL

ECM ANL ------------

GFS FORECAST

16

INPUT-----------OUTPUT( )

ECM(WF) Analysis------------------------- PSEUDO ECM “OBS”

PSEUDO ECM “OBS”GFS GUESS--------------

ECM ANLYSIS“PRE-COND” GUESS

Run GSI

Run GSIagain

“InterpGES Runs”

PRE-CONDECM GUESS

PSEUDO ECM “OBS”

---------------ECM ANL

ECM ANL ------------

GFS FORECAST3, 6, 9-hour

GFS FORECASTBKGND GES -3, 0, +3

GDAS Obs ------------

InterpGES Analysis

InterpGES Analysis

------------GFS

FORECAST5-days

17

Initialization Date GFS ECMWF ECM midlatOVRLY polarOVRLY ecmanlGES CNTRL interpGES2008011100 0.68 0.83 0.80 0.76 0.74 0.822008011212 0.69 0.89 0.86 0.83 0.82 0.772008020100 0.68 0.74 0.86 0.852008020300 0.65 0.83 0.83 0.82 0.69 0.632008021700 0.73 0.87 0.89 0.852008030300 0.67 0.82 0.62 0.612008030312 0.59 0.85 0.85 0.84 0.74 0.77 0.69 0.782008030912 0.71 0.91 0.81 0.78 0.69 0.79 0.802008031012 0.71 0.89 0.85 0.75 0.79 0.79 0.772008031300 0.69 0.71 0.83 0.792008031800 0.59 0.79 0.75 0.56 0.69 0.702008031812 0.66 0.63 0.76 0.69 0.78 0.66 0.78 0.752008032012 0.64 0.78 0.75 0.78 0.72 0.702008040900 0.69 0.85 0.84 0.762008042500 0.67 0.77 0.58 0.612008042512 0.67 0.80 0.72 0.72 0.63 0.692008042600 0.61 0.91 0.89 0.68 0.65 0.722008050900 0.68 0.90 0.82 0.802008051000 0.69 0.86 0.83 0.792008051012 0.84 0.90 0.90 0.892008051512 0.65 0.90 0.87 0.782008052200 0.60 0.87 0.84 0.83 0.73 0.702008052212 0.55 0.82 0.75 0.762008061212 0.64 0.85 0.88 0.832008062500 0.70 0.89 0.79 0.752008062512 0.66 0.87 0.77 0.70 0.72 0.772008072500 0.69 0.82 0.812008080500 0.69 0.87 0.892008081600 0.67 0.85 0.862008081912 0.68 0.84 0.82

Excerpt of SH Dropout Climatology List (AC skill scores)

NB: 90% ECM Success in alleviating SH dropout. Adding GDAS Obs (ecmanlGES) lowers the success rate to ~75%. GSI upgrades (CNTRL) improves GSI but not enough to alleviate dropouts, and for InterpGES runs also improves GSI but not enough to alleviate dropouts meaning that adding the SH observations (and the 3, 6, 9-h ges) degrade the ECM forecasts.

18

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

Initialization Date

Southern Hemisphere 5-day Anomaly Correlation Scores (Dropout Cases)

GFS

ECMWF

ECM

CNTRL

InterpECM

19

20

21

22

23

24

25

SH Dropout Composite Cntrl vs InterpGES

Zonal average (above) and mean composite difference between GFS Cntrl and InterpGES (Increments). GFS has low level SH warm “bias” if we consider ECMWF as ground truth.

SH CNTRL 5-day Composite AC = 0.71 SH InterpGES 5-day Composite AC = 0.73

26

Comparison of CNTRL vs. InterpGES Expt.

Dropout Date GFS ECMWF ECM CNTRL INTERPGES

2008042600 0.61 0.91 0.89 0.65 0.72

Guess Analysis

• InterpGES run introduces a lower height and temperature bias to GSI

• ANL differences are not significantly different after GSI interpolation but CNTRL maintains high bias

27

Impact of GSI and OBSThe addition of GSI Observations causes a positive height bias in the SH

CNTRL InterpGES

Dropout Date GFS ECMWF ECM CNTRL INTERPGES

2008042600 0.61 0.91 0.89 0.65 0.72

TypicalDropout case

28

Percent Better Draw for GDAS and ECM Analyses 300-200 hPa 12Z April 2008

0

10

20

30

40

50

60

70

80

90

EU

SA

TW

VA

DW

CA

ND

AR

SO

ND

ES

PR

OF

S

PIB

ALL

S

AIR

EP

AM

DA

R

AC

AR

S

NE

SD

ISW

VW

JMA

IRW

NE

SD

ISIR

W

MO

DIS

WV

WC

T

MO

DIS

WV

WD

L

MO

DIS

IRW

Data Type

Bet

ter

Dra

w in

%

GSI

ECM

EU Satwinds are not used in the GSI

• The GSI draws more formost Satwinds

29

Percent Better Draw for GDAS and ECM Analyses for all Radiosonde Winds 12Z April 2008

0

10

20

30

40

50

60

70

80

SFC-900

900-800 800-700 700-600 600-500 500-400 400-300 300-200 200-100 100-000

Pressure Category in hPa

Bet

ter

Dra

w in

%

GSI

ECM

The GSI draws more for sonde winds especially away from jet level

30

Percent Better Draw for GDAS and ECM Analyses for Radiosonde Winds with 5-10 m/sec Analysis Difs 12Z April 2008

0

10

20

30

40

50

60

70

80

90

100

SFC-900

900-800

800-700

700-600

600-500

500-400

400-300

300-200

200-100

100-000

Pressure Category in hPa

Bet

ter

Dra

w in

%

GSI

ECM

The GSI draws more for outlier winds – less so around jet level

31

Percent Better Draw for GDAS and ECM Analyses for all Radiosonde Temperatures 12Z April 2008

0

10

20

30

40

50

60

70

SFC-900

900-800 800-700 700-600 600-500 500-400 400-300 300-200 200-100 100-000

Pressure Category in hPa

Bet

ter

Dra

w in

%

GSI

ECM

The draw for all temperatures is similar – But see next slide

ECM fits better 300 hPa and up

32

Percent Better Draw for GDAS and ECM Analyses for Radiosonde Temperatures with 2-4 Degree Analysis Difs 12Z April 2008

0

10

20

30

40

50

60

70

80

90

SFC-900

900-800 800-700 700-600 600-500 500-400 400-300 300-200 200-100 100-000

Pressure Category in hPa

Bet

ter

Dra

w in

%

GSI

ECM

The GSI draws more for outlier temperatures in the middle atmosphere and the ECM draws more in the jet level to upper atmosphere

33

Satellite Wind Speed Biases OB-ANL Apr 2008 12Z ECM (Red), GSI (Blue) m/sec

JMA Type 245

-1.2 -0.7 -0.2 0.3

SFC-900

900-800

800-700

700-600

600-500

500-400

400-300

300-200

200-100

JMA IR Type 252

-1.2 -0.7 -0.2 0.3 0.8

700-600

600-500

500-400

400-300

300-200

200-100

MODIS IR Type 257

-1.2 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4

SFC-900

900-800

800-700

700-600

600-500

500-400

400-300

300-200

MODIS WV Type 259

-1.2 -0.7 -0.2 0.3

600-500

500-400

400-300

300-200

Note ECM biases are mostly negative compared to GSI, meaning ECM winds are stronger than GSI winds at satellite wind locations

34

Baroclinicity parameter for GFS and ECMWF

A suitable measure of the baroclinicity is given by the Eady growth rate maximum defined as:

σBI = 0.31 f |(-gp/RT) ∂V/ ∂p| N-1

where f is the Coriolis parameter, V is the total vector wind, N is the Brunt Väisällä frequency and all other parameters have their usual meaning.

This is done to show potential action areas or volatility to propagate initial condition errors into forecast differences

35

NH Eady Stability indexDifference: GFS - ECMWF

SH Eady Stability indexGFS - ECMWF

The predominance of Red over Blue shows GFS has more pronounced baroclinicity compared to ECMWF Ops in low levels (800mb)…

Largest differences for this NHDropout case, 20081021, are in Pacific (disregard most mountain areas as quantities are extrapolated values below Ground).

36

NH GFS Eady Stability index

NH ECMWF Eady Stability index

The total Eady-index shows the Greatest baroclinic potential in the eastern part of the broad Pacific trough, the differences in this case, cause a dropout in the 5-day

forecast.

37

NH Eady Stability indexGFS - ECMWF

SH Eady Stability index GFS - ECMWF

And at 500 mb and 200 mb (not shown), GFS has more pronounced baroclinicity compared to ECMWF Ops. The differences are as much as 20% of the total indexshown on next slide.

Large differences are along trough line with dipole, indicated by Blue and Red indicating difference in position (phase), and large potential for this Rossby wave details.