Georgia Institute of Technology
Comprehensive evaluation on air quality forecasting ability of Hi-Res
in southeastern United States
Yongtao Hu1, M. Talat Odman1, Michael E. Chang2 and Armistead G. Russell1 1School of Civil & Environmental Engineering,2Brook Byers Institute of Sustainable Systems
Georgia Institute of Technology
8th Annual CMAS Conference, October 19th, 2009
Georgia Institute of Technology
Outline
The Hi-Res air quality forecasting system.
2006-2009 O3 and PM2.5 performance for Atlanta metro.
Spatial variation of forecast performance.
Linking forecast performance to weather conditions.
Linking forecast performance to emissions conditions.
Georgia Institute of Technology
Hi-Res: forecasting ozone and PM2.5 at a 4-km resolution in metro Atlanta
area
CMAQ
SMOKE
Emission Inventory
WRF
NAM 84-hr Forecast
Forecast Product
MeteorologyEmissions
Air Quality
CMAQ
SMOKE
Emission Inventory
WRF
NAM 84-hr Forecast
Forecast Product CMAQCMAQ
SMOKE
Emission Inventory
SMOKESMOKE
Emission InventoryEmission Inventory
WRF
NAM 84-hr Forecast
WRFWRF
NAM 84-hr Forecast NAM 84-hr Forecast
Forecast Product Forecast Product
MeteorologyEmissions
Air Quality
Hi-Res Air Quality Forecasting SystemServing Metro-Atlanta Area since 2006
Hi-Res Modeling Domains
36-km (72x72)
4-km (99x78)
12-km (72x72)
Current 4-km (123x123)
36-km (72x72)
4-km (99x78)
12-km (72x72)
Current 4-km (123x123)
12Z 27Z
ramp up 36-12- & 4-km forecasts
77Z
00Z 15Z
ramp up 36-12- & 4-km forecasts
66Z
8pm 0am0am0am0amForecast Day 2
R&R&R&S F&R R
Forecast Day 1Forecast Day 2Forecast Day 2Forecast Day 2
S F
Cycle 1
Cycle 2
OperationForecast Day 1Forecast Day 1Forecast Day 1
12Z 27Z
ramp up 36-12- & 4-km forecasts
77Z12Z 27Z
ramp up 36-12- & 4-km forecasts
77Z
00Z 15Z
ramp up 36-12- & 4-km forecasts
66Z
8pm 0am0am0am0amForecast Day 2
R&R&R&S F&R R
Forecast Day 1Forecast Day 2Forecast Day 2Forecast Day 2
S F
Cycle 1
Cycle 2
OperationForecast Day 1Forecast Day 1Forecast Day 1
Hi-Res Cycle
Georgia Institute of Technology
Hi-Res Forecast Products“Single Value” Report: tomorrow’s AQI, ozone and PM2.5 by metro area in Georgia
Air Quality Forecasts: AQI, ozone and PM2.5, 48-hrs spatial plots and station profiles
Meteorological Forecasts: precipitation, temperature and winds, 48-hrs spatial plots and station profiles
Performance Evaluation: time series comparison and scatter plots for the previous daySnapshots from Hi-Res homepage:
http://forecast.ce.gatech.edu
Georgia Institute of Technology
Evolving history of Hi-Res during 2006-2009
Updated to new release of WRF each year before ozone season.
•WRF2.1, 2.2, 3.0 and 3.1
Projected NEI to current year in the very beginning of each year.
Updated forecast products website each year before ozone season.
Switched from single-cycle forecasting to two-cycles in 2008.
Enlarged 4-km domain to cover the entire Georgia in 2009.
Introduced Georgia Tech’s new SOA module in 2009.
Data assimilation in ozone forecasting is in experiment.
Georgia Institute of Technology
Purposes of Forecasting Performance Evaluation
To hopefully have a good performance show and hence to give a good reason for being further funded.
•It’s in fact very important…
To explore reasons for why bad forecasting so that we can improve.
•Science? Can we blame for “Smog Alert” that reduced emissions?
Finally for users of our forecasts: to build “quantitative” confidence in the forecasts.
•If today is a sunny Monday how I am going to trust their “Smog Alert”?
“Comprehensive” evaluation, preliminary results presented here.
Georgia Institute of Technology
Performance Metrics
N
kok
ok
mk
c
cc
N 1
1MNB
N
kok
ok
mk
c
cc
N 1
1MNE
False Alarms
Hits
Correct Nonevents
MissedExceedenc
es
For
ecas
t
Observation
NAAQS
NA
S
Georgia Institute of Technology
Overall 2006-2009 Performance: Atlanta Metro
Ozone PM2.5
MNB 17%
MNE 24%MNB -25%
MNE 37%
0
75
150
0 75 150
Obs.
4-km
94 98
34637
0.0
35.0
70.0
0 35 70
Obs.
4-km
0 0
517
47
Georgia Institute of Technology
Ozone Performance
Georgia Institute of Technology
Forecast vs. Observed O3
0.0
20.0
40.0
60.0
80.0
100.0
120.0
May-01 Jun-01 Jul-01 Aug-01 Sep-01
O3 (
pp
b)
Obs. 4-km
0
20
40
60
80
100
120
May-07 Jun-07 Jul-07 Aug-07 Sep-07
O3
(pp
b)
Obs 4-km
0
20
40
60
80
100
120
May-08 Jun-08 Jul-08 Aug-08 Sep-08
O3
(pp
b)
Obs 4-km
0
20
40
60
80
100
120
May-09 Jun-09 Jul-09 Aug-09 Sep-09
O3
(pp
b)
Obs 4-km
2006
20092008
2007
Georgia Institute of Technology
2006 O3 Performance: Hi-Res vs. EPD’s
0.0
85.0
170.0
0.0 85.0 170.0
Obs.
EP
D
9 17
10
117
0.0
85.0
170.0
0.0 85.0 170.0
Obs.
4-km
27 15
995
Our 4-km Forecast EPD Ensemble Forecast
MNB 11%
MNE 29%
MNB 6.2%
MNE 15%
Georgia Institute of Technology
2007 O3 Performance: Hi-Res vs. EPD’s
Our 4-km Forecast EPD Ensemble Forecast
MNB 8.5%
MNE 19%
MNB 9.0%
MNE 18%
0
85
170
0 85 170
Obs.
4-km
10 11
10611
0
85
170
0 85 170
Obs.E
PD
10 16
11210
Georgia Institute of Technology
2008 O3 Performance: Hi-Res vs. EPD’s
Our 4-km Forecast EPD Ensemble Forecast
MNB 17%
MNE 23%
MNB 11%
MNE 19%
0
75
150
0 75 150
Obs.
4-km
22 19
1039
0
75
150
0 75 150
Obs.E
PD
20 23
1065
Georgia Institute of Technology
2009 O3 Performance: Hi-Res vs. EPD’s
Our 4-km Forecast EPD Ensemble Forecast
MNB 28%
MNE 30%MNB 13%
MNE 21%
0
75
150
0 75 150
Obs.
4-km
24 7
1084
0
75
150
0 75 150
Obs.E
PD
7 6
1255
Georgia Institute of Technology
Particulate Matter Performance
Georgia Institute of Technology
Summer
Georgia Institute of Technology
Forecast vs. Observed PM2.5
0.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
40.0
45.0
May-01 Jun-01 Jul-01 Aug-01 Sep-01
PM
2.5
(m g
/m3)
Obs. 4-km
0
5
10
15
20
25
30
35
40
45
May-07 Jun-07 Jul-07 Aug-07 Sep-07
O3
(pp
b)
Obs 4-km
0
5
10
15
20
25
30
35
40
45
May-08 Jun-08 Jul-08 Aug-08 Sep-08
O3
(pp
b)
Obs 4-km
0
5
10
15
20
25
30
35
40
45
May-09 Jun-09 Jul-09 Aug-09 Sep-09
PM
2.5
(ug
/m3)
Obs 4-km
2006 2007
2008 2009
Georgia Institute of Technology
2007 PM2.5 Performance: 4-km vs. EPD’s
Our 4-km Forecast EPD Ensemble Forecast
MNB -37%
MNE 44%
MNB 8.6%
MNE 28%
0
35
70
0 35 70
Obs.
4-km
0 0
118
210
35
70
0 35 70
Obs.E
PD
9 12
11810
Georgia Institute of Technology
2008 PM2.5 Performance: 4-km vs. EPD’s
Our 4-km Forecast EPD Ensemble Forecast
MNB -38%
MNE 42%
MNB -0.2%
MNE 22%
0
35
70
0 35 70
Obs.
4-km
0 0
143
6
0
35
70
0 35 70
Obs.E
PD
2 2
1414
Georgia Institute of Technology
2009 PM2.5 Performance: 4-km vs. EPD’s
Our 4-km Forecast EPD Ensemble Forecast
MNB 8%
MNE 25%
MNB 11%
MNE 24%
0
35
70
0 35 70
Obs.
4-km
0 0
129
3
0
35
70
0 35 70
Obs.E
PD
0 0
129
3
Georgia Institute of Technology
Winter
Georgia Institute of Technology
Forecasted vs. Observed PM2.5
0
5
10
15
20
25
30
35
40
45
Oct-07 Nov-07 Dec-07 Jan-08 Feb-08 Mar-08
O3
(pp
b)
Obs 4-km
0
5
10
15
20
25
30
35
40
45
Oct-08 Nov-08 Dec-08 Jan-09 Feb-09 Mar-09
O3
(pp
b)
Obs 4-km
2007
2008
Georgia Institute of Technology
Spatial Variation of Performance: Ozone
30%
32%
26%Douglasville
29%
26%27%
28%37%
36%
26%
“Single value” forecast for Atlanta metro has a MNE as 24%
Georgia Institute of Technology
Spatial Variation of Performance: PM2.5
41%Douglasville
43%
40%
40%41%
39%
42%
“Single value” forecast for Atlanta metro has a MNE as 37%
Georgia Institute of Technology
Mostly Sunny
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0.00
%
10.0
0%
20.0
0%
30.0
0%
40.0
0%
50.0
0%
60.0
0%
70.0
0%
80.0
0%
90.0
0%
100.
00%
110.
00%
120.
00%
130.
00%
140.
00%
8-hr O3 Forecast Error
Rain
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0.00
%
10.0
0%
20.0
0%
30.0
0%
40.0
0%
50.0
0%
60.0
0%
70.0
0%
80.0
0%
90.0
0%
100.
00%
110.
00%
120.
00%
130.
00%
140.
00%
8-hr O3 Forecast Error
Cloudy
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
110%
120%
130%
140%
8-hr O3 Forecast Error
Partly Cloudy/Sunny
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0.00
%
10.0
0%
20.0
0%
30.0
0%
40.0
0%
50.0
0%
60.0
0%
70.0
0%
80.0
0%
90.0
0%
100.
00%
110.
00%
120.
00%
130.
00%
140.
00%
8-hr O3 Forecast Error
24%
24%24%
24%
78%64%
46%42%
This means a 78% chance that a 8-hr O3 forecasts error in a sunny day is less than 24%
Linking Performance to Weather Conditions: (1) Ozone
Georgia Institute of Technology
Mostly Sunny
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
110%
120%
130%
140%
24-hr PM2.5 Forecast Error
Rain
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
110%
120%
130%
140%
24-hr PM2.5 Forecast Error
Linking Performance to Weather Conditions: (2) PM2.5
Cloudy
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
110%
120%
130%
140%
24-hr PM2.5 Forecast Error
Partly Cloudy/Sunny
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
110%
120%
130%
140%
24-hr PM2.5 Forecast Error
37% 37%
37% 37%
49% 54%
67%53%
Georgia Institute of Technology
Linking Performance to Emissions Conditions: (1) Ozone
Monday and Friday
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
110%
120%
130%
140%
8-hr O3 Forecast Error
Tuesday,Wednesday and Thursday
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
110%
120%
130%
140%
8-hr O3 Forecast Error
Weekends and Holidays
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
110%
120%
130%
140%
8-hr O3 Forecast Error
24% 24%
24%
67%67%
67%
10% 10%
10%
30%35%
37%
Georgia Institute of Technology
Linking Performance to Emissions Conditions: (2) PM2.5
Monday and Friday
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
110%
120%
130%
140%
24-hr PM2.5 Forecast Error
Tuesday, Wednesday and Thursday
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
110%
120%
130%
140%
24-hr PM2.5 Forecast Error
Weekends and Holidays
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
110%
120%
130%
140%
24-hr PM2.5 Forecast Error
37%
37%
37%
53% 68%
39%
Georgia Institute of Technology
Summary
• 2006-2009 Ozone forecasts are good.– Overall bias is +17% and error is 24%
• 2006-2009 PM2.5 forecasts are not very accurate. – May-September bias is -25% and error is 37%
• The new SOA module helped a much better 2009 PM2.5 performance– May-September bias is 8% and error is 25%
• “Single Value” forecasts for Atlanta metro is slightly in better performance than specific station forecasts. – Larger spatial variance for ozone performance, PM2.5
performance is more uniform spatially.
• Less cloud coverage, better ozone performance, but worse PM2.5 performance
• Worse PM2.5 performance in weekends and holidays– But not seen for ozone performance.
Georgia Institute of Technology
Acknowledgements
We thank Georgia EPD for funding the Hi-Res forecasts, Dr. Jaemeen Baek of our group for the new SOA module,
Dr. Carlos Cardelino of Georgia Tech for team forecasts.