AN EVALUATION OF THE AN EVALUATION OF THE ETA-CMAQ AIR QUALITY FORECAST MODELETA-CMAQ AIR QUALITY FORECAST MODELAS PART OF NOAA’S NATIONAL PROGRAMAS PART OF NOAA’S NATIONAL PROGRAM

CMAQCMAQ

AIRNOWAIRNOW

Brian Eder*Brian Eder*Daiwen Kang * Daiwen Kang *

Ken Schere*Ken Schere*Robert Gilliam*Robert Gilliam*Jonathan Pleim*Jonathan Pleim*

Atmospheric Modeling DivisionAtmospheric Modeling DivisionAir Resources Laboratory, NOAAAir Resources Laboratory, NOAA

August 26,2003August 26,2003 * On assignment to NERL EPA* On assignment to NERL EPA

RTP, NC 27711RTP, NC 27711

Forecast ConfigurationForecast Configuration

- Eta Meteorology- CBIV Mechanism- SMOKE Emissions (Offline)- 12 km grid resolution - 22 Vertical Layers

48 Hr. Forecast (12Z Initialization)48 Hr. Forecast (12Z Initialization) 7 July – 31 September, 20037 July – 31 September, 2003 7 July – 31 August (shown)7 July – 31 August (shown)

48 Hr.48 Hr. Forecast (Corrected Land-use)Forecast (Corrected Land-use) 12 - 19 August12 - 19 August

Domain

Models-3 CMAQ

This evaluation used:

Hourly O3 concentrations (ppb) from EPA’s AIRNOW network

521 stations

7 July - 31 August

A suite of statistical metrics for both:

discrete forecasts and categorical forecasts

for the:

hourly, maximum 1-hr, maximum 8-hr O3 simulations

Two Forecast / Evaluation TypesTwo Forecast / Evaluation Types

- Discrete Forecasts Discrete Forecasts

[Observed] [Observed] versus versus [Forecast] [Forecast]

- Category Forecasts - Category Forecasts (Two Category) (Two Category)

Observed Exceedances, Non-ExceedancesObserved Exceedances, Non-Exceedancesversusversus

Forecast Exceedances, Non-ExceedancesForecast Exceedances, Non-Exceedances

Discrete Forecast / Evaluation Discrete Forecast / Evaluation StatisticsStatistics - Summary - Regression - Biases

- Errors

AIRNOW

M B M odel O bsN

N 1

1( )

NM BM odel O bs

O bs

N

N

( )

( )

1

1

100%

R M SE M odel O bsN

N

1 2

1

0 5

( ).

NM EM odel O bs

O bs

N

N

1

1

100%( )

[Observed] versus [Forecast]

Category Forecast / Evaluation Category Forecast / Evaluation

- Two Category Forecasts - Two Category Forecasts

Observed Exceedances, Non-ExceedancesObserved Exceedances, Non-Exceedances

versusversus

Forecast Exceedances, Non-ExceedancesForecast Exceedances, Non-Exceedances

a b

c d

Fore

cast

Exc

eeda

nce

N

o

Ye

s

No YesObserved Exceedance

a ba b

c dc d

Category ForecastCategory ForecastAccuracy Percent of forecasts that correctly predict event or non-event.

Bias Indicates if forecasts are under-predicted (false negatives) or over-predicted (false positives)

False Alarm Rate

Percent of times a forecast of high ozone did not occur

Ab c

a b c d%

100

Ba bb d

a ba b

c dc d

FARa

a b%

100

Critical Success Index

How well the high ozone events were predicted.

Probability Of Detection Ability to predict high ozone events

CSIb

a b d

100%

Category ForecastCategory Forecasta ba b

c dc d

PO Db

b d%

100

a

c

a ba b

c d c d a= 151b= 1c= 24,227d= 4n= 24,383

CMAQ = 34.9 + 0.65(AIRNOW)

Max 1-hr OMax 1-hr O33

7 July – 31 August

Summary Statistics

Discrete Evaluation

Categorical Evaluation

[ppb] CMAQ AIRNOW CMAQ = 34.9 + 0.65 (AIRNOW)

Ozone 125 ppb

Mean 71.6 56.5 r 0.60 A 99.4%

SD 18.1 16.6 n 24,383 B 25.5

CV 25.2 29.3Max 182.9 132 BIASES

95th 103.4 84 MB 15.1 FAR 99.4%75th 83.0 67 NMB 26.9% CSI 0.6%

50th 70.5 56

25th 58.0 45 ERRORS

5th 45.4 30 RMSE 21.9 POD 16.7%

Min 0 1 NME 31.7%

Max 1- hr OMax 1- hr O33

Temporal EvaluationTemporal Evaluation

– – Max 1 hr OMax 1 hr O33

7 July

1 August

31 August

-92 -90 -88 -86 -84 -82 -80 -78 -76 -74 -72 -70 -68

34

36

38

40

42

44

46

48

-0.5 to 0 .25 0.25 to 0.5 0.5 to 0 .75 0.75 to 4

Spatial Evaluation Spatial Evaluation

Max 1- hr OMax 1- hr O33CorrelationCorrelation

0.00 – 0.250.25 – 0.500.50 – 0.750.75 – 1.00

Mean = 0.60

-92 -90 -88 -86 -84 -82 -80 -78 -76 -74 -72 -70 -68

34

36

38

40

42

44

46

48

-1 4 to 10 1 0 to 20 2 0 to 30 3 0 to 40 4 0 to 50

Max 1- hr OMax 1- hr O33Mean BiasMean Bias

Spatial Evaluation Spatial Evaluation

-10 – 10 10 - 20 20 - 30 30 – 40 40 – 50

Mean = 15.1

Spatial Evaluation Spatial Evaluation

Max 1- hr OMax 1- hr O33Root Mean Square ErrorRoot Mean Square Error

-92 -90 -88 -86 -84 -82 -80 -78 -76 -74 -72 -70 -68

34

36

38

40

42

44

46

48

0 to 10 1 0 to 20 2 0 to 30 3 0 to 40 4 0 to 50

0 – 10 10 - 20 20 - 30 30 – 40 40 – 50

Mean = 21.9

CMAQ = 35.5 + 0.64(AIRNOW)

a ba b

c d c d a= 3276b= 149c= 20,979d= 65n= 24,469

Max 8-hr OMax 8-hr O33

Summary Statistics

Discrete Evaluation

Categorical Evaluation

[ppb] CMAQ AIRNOW CMAQ = 35.5 + 0.64 (AIRNOW)

Ozone 85 ppb

Mean 67.2 49.6 r 0.57 A 86.3%

SD 16.5 14.7 n 24,469 B 16.0

CV 24.5% 29.7%

Max 162.2 108.4 BIASES

95th 95.8 73.9 MB 17.6 FAR 95.6%75th 78.0 59.8 NMB 35.8% CSI 4.2%

50th 66.4 49.5

25th 54.7 39.1 ERRORS

5th 43.3 26.1 RMSE 23.0 POD 69.6%

Min 0 1 NME 39.1%

Max 8- hr OMax 8- hr O33

Temporal EvaluationTemporal Evaluation – – Max 8 hr OMax 8 hr O33

1 August

31 August

7 July

Spatial Evaluation Spatial Evaluation

Max 8- hr OMax 8- hr O33CorrelationCorrelation

-92 -90 -88 -86 -84 -82 -80 -78 -76 -74 -72 -70 -68

34

36

38

40

42

44

46

48

-0 .0 5 to 0 .25 0.25 to 0.5 0.5 to 0 .75 0.75 to 1

0.00 – 0.250.25 – 0.500.50 – 0.750.75 – 1.00

Mean = 0.57

Spatial Evaluation Spatial Evaluation

Max 8- hr OMax 8- hr O33Mean BiasMean Bias

-92 -90 -88 -86 -84 -82 -80 -78 -76 -74 -72 -70 -68

34

36

38

40

42

44

46

48

-1 2 to 10 1 0 to 20 2 0 to 30 3 0 to 40 4 0 to 50

-10 – 10 10 - 20 20 - 30 30 – 40 40 – 50

Mean = 17.6

Spatial Evaluation Spatial Evaluation

Max-8 hr OMax-8 hr O33Root Mean Square ErrorRoot Mean Square Error

-92 -90 -88 -86 -84 -82 -80 -78 -76 -74 -72 -70 -68

34

36

38

40

42

44

46

48

0 to 10 1 0 to 20 2 0 to 30 3 0 to 40 4 0 to 50

0 – 10 10 - 20 20 - 30 30 – 40 40 – 50

Mean = 23.0

Land-Use ErrorLand-Use Error

Land-use fields associated with Eta were being post-processed Land-use fields associated with Eta were being post-processed incorrectly. As a resultincorrectly. As a result::

- - Most of the domain was classified as water.Most of the domain was classified as water.- Dry deposition was greatly under simulated- Dry deposition was greatly under simulated

This error was discovered/corrected by NCEP on Sept. 9This error was discovered/corrected by NCEP on Sept. 9 thth..

- An eight day period (12-19 August) was re-simulated.- An eight day period (12-19 August) was re-simulated.- Positive biases were cut in half, errors reduced also.- Positive biases were cut in half, errors reduced also.

Run rMB

(ppb)NMB(%)

RMSE(ppb)

NME(%)

A(%)

B FAR(%)

CSI(%)

POD(%)

Initial 0.64 16.2 27.5 23.0 31.7 99.0 - 100.0 0.0 -

Corrected 0.66 7.6 13.0 16.6 21.7 99.6 - 100.0 0.0 -

Max 1-hr O3

Max 8-hr O3

Comparison BetweenInitial and Corrected Simulations

August 12 –19 2003

Run rMB

(ppb)NMB(%)

RMSE(ppb)

NME(%)

A(%)

B FAR(%)

CSI(%)

POD(%)

Initial 0.62 19.2 37.2 24.6 39.9 76.2 - 100.0 0.0 -

Corrected 0.64 10.4 20.1 17.1 26.3 90.7 3.5 92.0 6.6 28.0

Temporal Evaluation Temporal Evaluation (Corrected August 12 –19)(Corrected August 12 –19)

– Max 1 hr O – Max 1 hr O3 3

– Max 8 hr O– Max 8 hr O33

SummarySummaryThe Eta-CMAQ modeling system performed reasonably well, in this, its first attempt at forecasting ozone concentrations:

Correlation: 0.57 - 0.60 Bias: 15.1 ppb (26.9%) - 17.6 ppb (31.7%) Error: 21.9 ppb (31.7%) - 23.0 ppb (39.1%) Accuracy: 86.3 - 99.4%

An error was discovered in Eta’s post processed land-use designation that resulted in the:

– under-estimation of dry deposition and – hence over-simulation of concentrations

Once corrected, the positive biases and errors were greatly reduced:

Correlation: 0.64 - 0.66 Bias: 7.6 ppb (13.0%) - 10.4 ppb (20.1%)

Error: 16.6 ppb (21.7%) - 17.1 ppb (26.3%) Accuracy: 90.7 - 99.6%

Contact:Contact:

Brian K. Eder

email: eder@hpcc.epa.gov