WMO Aircraft Lead Center Work

Dr. Bradley BallishNOAA/NWS/NCEP/EMC/GCWMB

12 April 2013

“Where America’s Climate and Weather Services Begin”

Outline • Aircraft data are important for model forecast skill,

but there have been many challenging problems to solve involving help from many people of different countries

• Aircraft data coverage is getting better• NCEP monthly aircraft data monitoring reports• Special monitoring reports and data alerts• Colleagues supporting aircraft data efforts• Data quality and QC problems• Aircraft temperature bias problems and correction• Reject-list optimization• Fast alerts for data problems• Conclusions and the way forward

Impact of GOS components on 24-h ECMWF Global Forecast skill(courtesy of Erik Andersson, ECMWF)

Forecast error reduction contribution (%)

AIREP denotes all aircraftdata and has large errorreduction even though thecoverage is not that goodcompared to satellite data

AMSUA, IASA and AIRS satellite data have bigger impact than aircraftpartly due to their global coverage.Radiosonde (TEMP) data have less impact than aircraft

Aircraft data are important impartbecause of their quality, manyproblems getting fixed and fromexpanding data coverage

But Aircraft Coverage Could be Much Better and is Expanding

• The following data coverage slides from the Navy Research Lab website http://www.usgodae.org/cgi-bin/cvrg_con.cgi, courtesy of Dr. Patricia Pauley*, shows that although aircraft coverage has increased, it still has large areas with no data– These slides are for 18Z for just one case, but other times and days

will show similar large data voids– Currently some Automatic Dependent Surveillance (ADS) data are

being put on the GTS in the North Atlantic, see green dots 3 slides later, but there could be much more if all the ADS data were on the GTS

* Patricia Pauley has made many contributions to analyzing aircraft data problems and their quality control

Canadian AMDARdata (red dots)are not yet used in the GFS

Green dotsshow ADSreports

Automatic Dependent Surveillance (ADS) reports are going to aircrafttraffic control at many places in the world, but little is on the GTS

These reportsmay be due toADS reports dueto high frequency

This side shows that the MDCRS moisture data coverage is getting better,with blue around cruise level and red-orange at lower levels

Location of Southwest ACARS Moisture Reports in NCEP GDASrun for 18Z 15 November 2012They have better quality than sonde moisture data– coverage is getting better

We also haveUPS moisturedata, but notas much

Data Coverage Conclusions• Since aircraft data are important for model

forecast skill, getting more and better aircraft data coverage will help forecasts and the airlines

• The WMO and ICAO are working on getting ADS data on the GTS but there are interagency disagreements

• If more ADS data are on the GTS, centers like NCEP could report on units with data problems, such as temperatures that are too warm which may impact fuel economy

• The NWS and or WMO are working on getting Hawaiian ACARS and Mexican AMDAR data and more ACARS moisture data

NCEP Monthly Data Monitoring Reports

• For the US MDCRS data, also referred to as ACARS data, we have the latest 12 months of statistics versus the NCEP model background at website: http://www.nco.ncep.noaa.gov/pmb/qap/acars/

• Non-US AMDAR data reports are at: http://www.nco.ncep.noaa.gov/pmb/qap/amdar/

• A track-checking code is run on the whole month’s data to diagnose problems like position errors and gross or stuck data

• The above AMDAR reports and reports to the US airlines contain easy to understand detailed explanations of data problems

Special Monitoring Reports and Alerts

• As the key person for the lead center on aircraft data, I have tried to be the leader in issuing many special reports and alerts on data problems

• However, people from other centers sometimes first find problems, such as:– In November 2009 Colin Parrett first reported on slow speed

biases with the new Chinese AMDAR data as described later– These reports and alerts are very important for NWP centers to

take correction action and also to get the problems fixed

• The next slides give a partial list of people and their country or agency that have been involved with finding, analyzing or fixing many aircraft data problems or helping expand data coverage

Colleagues Involved in Finding, Analyzing and Fixing Aircraft Data Problems

• Australia: Jeff Stickland, Dean Lockett, Mike Berechree, Kelvin Wong and Doug Body• ECMWF: Francois Lalaurette, Antonio Garcia, Erik Andersson, Dick Dee, Lars

Isaksen and Drasko Vasiljevic• UKMET: Colin Parrett• European AMDAR: Stewart Taylor and Stig Carlberg • South Africa: Gaborekwe Khambule, Mike Edwards, Kobus Olivier and others• Japan: Kazutoshi Onogi, Junichi Ishida and others• Netherlands: Jitze van der Meulen and Frank Grooters• Germany: Clemens Drüe and Axel Hoff• France: Herve Benichou• Canada: Charles Anderson, Gilles Verner, Yulia Zaitseva and Gilles Fournier• China: Xiang Li, Jiangling Xu and others• US military: Patricia Pauley, Doug Stewart and Eric Wise• US NOAA: William Moninger*, David Helms, George Schmidt, Stan Benjamin, Krishna

Kumar, Curtis Marshall, Carl Weiss and others• ARINC: Al Homans, Alan Williard and Jeannine Hendricks• US airlines: Many people including Carl Knable, Randy Baker and Rick Curtis

* Bill Moninger has contributed many papers and reports on aircraft data, and his website is very useful: http://amdar.noaa.gov/java/

Data Quality and QC Problem Examples

• The following group of slides show or discuss various problems with aircraft data quality

• These include:– Stuck (spuriously constant data) – Groups of aircraft with wrong locations– Aircraft with temperatures off by a factor of ten– Old data sent as current– New data may have problems

• There have been many more and different serious problems with aircraft data in the past that are not well documented or may not even have been discovered since monitoring tools are much better today than years ago

• These past problems need to be corrected for model climate studies of the past

14

Green means passes Red means ACQC deletesBlack SDM deletes Blue VARQC deletes

NCEP GDAS OB – BG Temperature Increments for Aircraft EU4264with stuck temperatures and altitude 9-10 February 2013

All reports were at exactly 641.7 hPa!

Very few reports were rejected by aircraft QCas it does not use background values instuck data QC decisions

SDM deleted tempsbut not winds

Stuck (spuriously constant data) is a common problem for aircraft and marine reports

A bigger QCtime-windowwould help

15

EU4264 report increments shown here are at spurious 641.7 hPa

-1000The GDAS had little impact from these bad winds, but the NAM had impact

16

17

NAM may havemore impact dueto no VARQC anda smaller time-window

Wind increments in knots to background for winds passing NCEP GSI QC in knotsfor spurious wind directions stuck at 360 degrees6 to 20 July 2009 for ACARS unit X1B2MIRAMany more winds failed operational QCOnly 12 of these passed NRL QC, but that is too many

A bigger QCtime-windowwould help

A fast dataalert systemwould help

For a period of a few years, the South African AMDAR data hadgroups of reports at wrong locations like group 2 this example. No center’s QCcould handle this, so NCEP had them on the reject-list for a long time

2

1

3

The bluenumbers arevector winddifferences tothe backgroundin knots

Grouptrack-checkerror example

My aircraft QCcode hassegment QCchecks

1

2

3

AU0137 Locations and Winc 06Z June 2009

Wind increments to background (Winc)are in knots and shown in red.Reports in group 2 are all in the wronglocation which is a very difficult QCproblem for most NWP centers

Australian aircraftdeveloped thesame problem.Dean Lockettof Australian thenhelped fix the sameAfrican problem

For a period of a few months, some European AMDAR reportswere bouncing off the equator until a fix was made

All of the spuriousreports in the leftside of this V patternpassed the NRLACQCin the NCEP GDAS

Stewart Taylor took actions to get this problem fixed

From 5 to 10 November 2009, there were large spurious impacts onthe NCEP GDAS analysis due to new Chinese AMDAR reports with windsin m/sec versus expected knots – If aircraft QC had a history file andlogic to delete suspect new data, this could have been prevented

Winds in knots

Internationalemailstriggered byColin Parrett’sfindings ledto a fix in days

Aircraft Temperatures with a Factor of 10 Error

• For a period from May 2011 to September 2012, typically a few European AMDAR aircraft per month had reports with temperatures that appeared to be one tenth of the correct value such as:– If the measured T was -50.3, the report would be -5.0– If the measured T was 4.8, the report was 0.5

• When this happened and the true temperature was in a range of roughly -5.0 to +5.0, then the reported error is in a range of -4.5 to 4.5 and the data would sometimes pass NCEP QC and impact the analysis a few degrees in some cases over small areas near airports

• Again this is a case where better QC logic could delete the bad data – for reanalyses, these should be deleted

• Stewart Taylor of the EU AMDAR program would shutoff such data until the problem was fixed when informed of the problem

Old Data Transmitted as Current• In some past cases, hundreds of past aircraft

reports were transmitted by mistake as current data– Fortunately, this happens rarely, such as twice per

year or less– Codes that can find this sort of problem are available

but not in operations– Next slide shows a case with day-old Australian

AMDAR data, where NCEP SDM QC actions coupled with the VARQC prevented bad impact

– Kelvin Wong of Australia made a quick fix for this

• There are more cases with small amounts of old ACARS transmitted as current– These often have no or small impact, but two slides later shows

a case with NAM impact

In this case, several hundred day-old Australian AMDAR reports weresent out as current when after a data server went down. NCEP analysisimpact was small helped by SDM QC actions

These aircraft wind incrementsalong the California coast arelarge as they were duplicatesof the previous day’s data

Next slide shows analysisminus background changesthat were large in NAM.They were small in GFS runwhere either the aircraft QC orVARQC deleted the data

Pressuresaround670 hPafrom ACARSunit P0IKJHBA

NAM analysis is impactedby spurious 20 knot vector wind changes

Temperature Bias Problems

• Aircraft with excessive temperature biases has been the most common problem for many years

• In addition, the following group of slides show that aircraft and sondes have different temperature biases

• The aircraft biases vary with:– Aircraft types– Phases of flight such as ascent, descent and level– Pressure – Individual aircraft of the same type– Biases can also change suddenly with time

• Sonde temperature biases vary with sonde types, pressure and different solar angles

• GPSRO data can help derive the bias corrections

Aircraft vs Sonde NCEP GSI Draws to Temps between 200-300 mb

# Aircraft >> # Sondes, thus warm aircraft data overwhelms the GSI/GFS system and the model is not truth, Nov 2008

Aircraft Tdiff (obs-ges)

Aircraft Tdiff (obs-anl)SOND Tdiff (obs-anl)

SOND Tdiff (obs-ges)

Temperature differences in tenths of a degree C

From:Sun, B., A. Reale, S. Schroeder,D. Seidel and B. Ballish (2013),Towards improved correction forradiation-induced biases inradiososonde temperatureobservations, accepted inJ. Geophys. Res.

Based on collocations withGPSRO data, the US sondesare too cold around 400-200 hPaso bias corrections should nottreat sondes or the backgroundas truth

RADCOR is needed for sondesand bias corrections foraircraft temperatures

-1.4

-1.2

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.675

7-22

2

757-

223

757-

24A

PF

737-

3H4

757-

232

767-

34A

F

A30

0F4-

60

737-

522

MD

-11F

737-

832

MD

-88

757-

251

A31

0-20

3

767-

332

A31

0-32

4

Bia

s C

orre

ctio

ns in

Deg

rees

C

Aircraft Types

Proposed ACARS Temperature Bias Corrections January 2007

SFC-700 700-500 500-300 300-150

From Ballish and Kumar BAMS(Nov 2008)

These corrections assumed the sondes were correct, which is not always true

Biases alsovary withascent ordescent

Reject-list Optimization

• Data with bad stats or problems should be added to the reject-list to prevent negative impact on the analyses

• When are stats bad enough that data should be on the reject-list? – It would be good to have a system where adjoints of

the analysis and forecast model could estimate impact forecast skill depending on reject-list criterion

• How quickly can we add or remove data from the reject-list? – The next few slides introduce a new fast alert system

for aircraft data developed at NCEP

33

Bad Data Alert Code System

• The current version of the new codes produce daily stats on aircraft temperatures and winds in 3 pressure levels– This includes biases and RMS differences to the model background plus counts

of total, rejected and gross observations– There are also wind direction stats for one deep level

• Another code then takes the latest 7 days worth of daily stats and averages the stats going backward one day at a time from the latest day– If any of these stats have a combination of bad enough quality plus enough

observations, alerts are made– Alerts can be to add or remove the aircraft from the reject-list– The daily stats in the latest history file are easy to view for checking on the alert

decisions– So far, the current code is working well but needs more work

34

Future Updates to Alert Code System

• Aircraft track-checking will be run first to:– Check for counts of track-check errors and stuck data problems– Allow for effective thinning of the data so that aircraft with very high

reporting rates are not treated as statistically independent data

• Checks on track-check errors and stuck data rates will be added to the alert codes

• Moisture stats will be added• More tuning will be performed on the alert logic which is a

function of the data quality and counts• Eventually ARINC could use the alerts for flagging the

MCDRS data• The system needs to be automated with results shared with

other centers and corrective action performed on the problems

-1

0

1

2

3

4

5

6

3 7 8 9 10 14 15 16 17 18 21

Day of Month in February 2012

Daily Temperature Biases for Aircraft JOHWUUBA February 2012

LOW

MID

HIGH

New code alertedearly on the 17th

Conclusions and the Way Forward• Aircraft data are important for model forecast skill even

though their coverage is not as good as desired– The data coverage is better and growing– More aircraft are getting moisture sensors– Lead center work supported by many international colleagues is

helping to quickly find data and QC problems and fixes, which helps make aircraft data more important

• Faster alerts are needed to be shared internationally for data problems

• More sharing of information on optimal quality control, bias correction and analysis use of aircraft data is needed

• Continue feedback to data providers to get problems fixed

• Thorough deletion of problem aircraft data in climate runs